Analyzing Weak Spikes via Cycle Counts: Estimation, Testing, and Computation

Large datasets play a central role in advancing fields such as genomics, signal processing, and the social sciences. These datasets often share a common structure: a meaningful but hidden signal masked by complex and unpredictable noise. Extracting this signal is a fundamental challenge across disciplines. In many modern applications, however, the signal is weak, and the noise is heterogeneous or poorly understood, which limits the effectiveness of existing methods. This project develops a new statistical framework, together with scalable AI-assisted algorithms, to reliably infer weak signals from large, noisy data without requiring prior knowledge of the noise structure. By developing new statistical methods and advancing theory for complex data structures, this project strengthens the reliability of statistical inference and promotes scientific innovation across applications. It will improve the analysis of biomedical and genomic data, contributing to advances in national health. Furthermore, the framework extends to large-scale social networks, supporting public policy design. Beyond research, the project will support STEM workforce development by mentoring graduate students, integrating modern statistical methods into the curriculum, and providing targeted training activities, such as a short course, to early-career scholars. This project studies inference in spiked models, which represent observed data as a low-rank signal matrix (spikes) corrupted by noise. The focus is on a challenging regime with weak signals and unknown, heterogeneous noise variance profiles. In this setting, the signal strength is comparable to the noise level, and the noise may vary across entries, rendering standard approaches based on spectral methods and random matrix theory ineffective. To address these challenges, the project introduces a novel framework based on cycle count statistics, which aggregate information across the data matrix and adapt to both unknown noise structures and weak signals. The research pursues three objectives. First, the investigator will develop new estimators for weak signals under general noise conditions by leveraging the properties of cycle count statistics and algebraic identities. Second, the investigator will establish hypothesis tests for signal detection under broad model assumptions. Third, the investigator will design AI-assisted graph-theoretic algorithms to construct computationally efficient proxy statistics that approximate cycle count statistics at scale. Together, these advances provide new theoretical insights into high-dimensional inference in spiked models, establish a flexible moment-based framework for signal estimation and detection, and enable practical implementation in large-scale applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.