Collaborative Research: FRGMS: Exploring hidden components in complex data: Geometry, uncertain quantifications, and dynamics

The explosion of relational data, such as health records, social networks, and ranking systems, presents both opportunities and challenges for advancing science and addressing societal needs. These datasets often reveal hidden structures that can be represented using mathematical tools like non-negative matrices or tensors. However, current approaches face significant limitations: they struggle to account for the varied nature and complexity of real-world data, the relationships between key quantities and underlying structures are often unclear, and the data may evolve over time, exhibit hierarchical patterns, or be sparse, making analysis even more difficult. This project aims to overcome these challenges by developing innovative methods and theories to better understand and analyze complex datasets. The research will create new tools for uncovering hidden patterns in data, advance mathematical theory for analyzing highly heterogeneous datasets, and apply these insights to improve large language models (LLMs). The results will include publicly available software, new datasets, and methodologies that will empower scientists and professionals across fields to address pressing questions in health, business, and beyond. The project also supports education by training students in modern statistical and machine learning techniques and fostering professional networks, aligning with NSF’s mission to advance science, education, and societal well-being. Modern datasets, including text, network, and relational data, often exhibit significant complexity, severe heterogeneity, and sparse signals, posing challenges for conventional analytical methods. This project addresses these challenges using a unified signal-plus-noise framework, where the signal is represented as a non-negative matrix or tensor (hidden components), and the noise is highly heterogeneous. The research has three main objectives. The first objective is to explore and leverage a low-dimensional simplex structure discovered in the spectral domain to develop novel tools for fitting hierarchical topic models and dynamic network models. The second objective is to advance random matrix theory (RMT) for such models by deriving sharp entry-wise bounds and uncertainty quantifications for leading eigenvectors in severely heterogeneous settings. These theoretical innovations will guide the development of optimal methods for analyzing complex datasets. The third objective is to generate new text datasets and use them to develop innovative language models capable of predicting influential citations and detecting flaws in financial statements. These methods and theoretical contributions will advance our ability to analyze and interpret complex datasets, enabling new applications across science and industry. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.