Adaptive Inference by Stabilized Cross-Validation

Modern data analysis and statistical learning are characterized by two defining features: complex data structures and black-box algorithms. The complexity of data structures arises from advanced data collection technologies and data-sharing infrastructures, such as imaging, remote sensing, wearable devices, and genomic sequencing. In parallel, black-box algorithms—particularly those stemming from advances in deep neural networks—have demonstrated remarkable success on modern datasets. This confluence of complex data and opaque models introduces new challenges for uncertainty quantification and statistical inference, a problem we refer to as ``black-box inference''. This research project aims to develop flexible, valid inference procedures for modern complex data that harness the strengths of black-box machine learning algorithms. These contributions have potential applications in areas such as policy evaluation, model selection, treatment effect identification, and algorithmic fairness auditing. A central focus of the project is the development of novel variants of a classical statistical tool: cross-validation, repurposed to enable adaptive inference in conjunction with powerful black-box models. Although cross-validation is widely used for evaluating estimator performance, its theoretical foundations remain limited, particularly in the context of complex data and modern algorithms. This research will begin with a multi-population comparison problem, using a stabilized cross-validation framework, and will then investigate performance guarantees of cross-validation in more general settings. The project will also develop new methods for adaptive population comparisons in high-dimensional and nonparametric regimes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.