CAREER: Distribution-Free and Adaptive Statistical Inference

Recent years have witnessed a growing trend across scientific disciplines to embrace complex modeling and black-box machine learning algorithms. Despite the remarkable success of handling complex data structures and fitting sophisticated regression functions, there remains a substantial gap regarding the integration of rigorous statistical principles into these pipelines. The main difficulty revolves around achieving reliable uncertainty quantification and robust statistical inference without artificially simplifying the complexity inherent in these advanced tools. Most existing frameworks that aim to bridge the gap rely on strong assumptions under which the machine learning algorithm can accurately estimate the data generating distribution. Nevertheless, these assumptions are often hard to justify, especially for modern machine learning algorithms that have yet to be fully understood. This research project aims to develop new frameworks for statistical inference that wrap around any machine learning algorithms or complex models without concerning about failure modes. The resulting methods are able to address the potential threats to inferential validity caused by black-box machine learning algorithms in a wide range of applied fields, including medicine, healthcare, economics, political science, epidemiology, and climate sciences. Open source software will also be developed to help applied researchers integrate rigorous statistical inference into their domain-specific modeling workflows without compromising the effectiveness of modern tools in non-inferential tasks. This may further alleviate hesitation in adopting modern machine learning methods and catalyze collaboration between scientific and engineering fields. Throughout the project, the PI will mentor undergraduate and graduate students, equipping them with solid understandings of statistical principles to become future leaders in face of rapidly evolving machine learning techniques. This proposal will focus on distribution-free inference, which is immune to misspecification of parametric models, violation of nonparametric assumptions like smoothness or shape constraints, inaccuracy of asymptotic approximations due to limited sample size, high dimensionality, boundary cases, or irregularity. To avoid making uninformative decisions, an ideal distribution-free inference framework should also be adaptive to good modeling. This means that it should be as efficient as other frameworks that rely on distributional assumptions. Adaptivity alleviates the tradeoff between robustness and efficiency. The PI will develop distribution-free and adaptive inference frameworks for three specific problems. First, in causal inference, tighter identified set can be obtained for partially identified causal effects by incorporating pre-treatment covariates. However, existing frameworks for sharp inference require estimating conditional distributions of potential outcomes given covariates. The PI will develop a generic framework based on duality theory that is able to wrap around any estimates of conditional distributions and make distribution-free and adaptive inference. Second, many target parameters in medicine, political economy, and causal inference can be formulated through extremums of the conditional expectation of an outcome given covariates. In contrast to classical methods that impose distributional assumptions to enable consistent estimation of the conditional expectation, the PI will develop a distribution-free framework for testing statistical null hypotheses and constructing valid confidence intervals on the extremums directly. Finally, the use of complex models and prediction algorithms in time series nowcasting and forecasting presents challenges for reliable uncertainty quantification. To address this, the PI will develop a framework based on model predictive control and conformal prediction that is able to wrap around any forecasting algorithms and calibrate it to achieve long-term coverage, without any assumptions on the distribution of the time series. The ultimate goal of this research is to bring insights and present a suite of tools to empower statistical reasoning with machine learning and augment machine learning with statistical reasoning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.