Hunt-and-Test Procedures: Design and Derandomization

This project addresses a fundamental challenge in testing statistical hypotheses: reliably detecting signals from complex data while avoiding false discoveries due to "double dipping", a practice of unintentionally using the same data to both identify and test hypotheses. Double dipping inflates the rate of false findings, thereby undermining the credibility and replicability of scientific research across various fields, including health, economics, and political science. To overcome this problem, this project develops innovative statistical methods known as "hunt-and-test" procedures, which adaptively seek out meaningful signals in data and rigorously validate them without bias. By introducing novel derandomization techniques, the research also eliminates the randomness and variability inherent in current methods, thereby significantly enhancing the replicability of results. These improved methods will directly support the reliable analysis of scientific data and foster greater public trust in research outcomes. Additionally, the project supports educational advancement by training graduate and undergraduate students in the research project. This project introduces a novel framework for constructing data-adaptive statistical tests through carefully designed and derandomized hunt-and-test procedures. "Hunt and test" randomly splits iid data into A and B, first using A to identify potential signal and then validating the signal with B, thereby maintaining rigorous calibration. However, this approach traditionally suffers from reduced statistical power and replicability due to the random data splitting. This research addresses these limitations by aggregating multiple replicates of the hunt-and-test procedure and recalibrating them using advanced techniques, such as rank-transformed subsampling —a method recently developed by the investigator. The project's two primary aims are to enhance both the robustness and efficiency of this framework. Aim I develops a hunt-and-test strategy to extend the classical score test to accommodate infinite-dimensional parameters, such as regression functions and log-densities, while flexibly integrating machine learning for powerful signal detection and ensuring calibration via debiasing. Aim II advances derandomization methods, optimizing their computational speed and statistical power through low-order approximation techniques and adaptive data aggregation. Together, these innovations substantially improve the performance of statistical hypothesis testing, broadening its applicability to complex, semiparametric models and a variety of scientific disciplines. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.