Collaborative Research: MFAI: Two Sides of a Tapestry: Causal Inference and Model Discovery Amid Information Gaps in Complex Data

This project aims to advance machine learning methods for discovering cause-and-effect relationships in complex systems. While much of modern data science focuses on identifying patterns and correlations in data, such associations cannot explain why events happen or how changing one factor might influence another. Causal discovery addresses this fundamental challenge by revealing the mechanisms behind observed phenomena, enabling more informed decisions, reliable predictions, and targeted interventions across fields such as healthcare, economics, engineering, and public policy. Despite recent AI advancements, determining causality from complex, large, noisy or incomplete datasets remains challenging. This research tackles that challenge by developing new theoretical models and analytical tools that target both specific causal inference and broader causal structure discovery. By integrating approaches from statistics, computer science, and mathematics, this work seeks to create AI systems that are more transparent, interpretable, and scientifically grounded. The anticipated outcomes are expected to significantly advance multiple fields by fostering interdisciplinary collaborations and paving the way for future discoveries in causality and data-driven problem-solving. To address the challenges of causal discovery and inference in the presence of missing, incomplete, or limited data, the project is organized around three closely connected research thrusts: (1) Causal Inference in the Presence of Unmeasured Confounders, which will focus on identifying causal effects as functions of observed data and estimating them robustly in the presence of hidden variables; (2) Differentiable Causal Graph Learning from Partially Observed Data, which will develop scalable, optimization-based methods for learning causal structures when data are noisy or partially missing; and (3) Causal Inference and Modeling Amid Insufficient Data via Large Language Models (LLMs), which will leverage the vast scientific knowledge embedded in literatures and databases to guide discovery when observational data are sparse. The proposed LLM-powered framework will extract relevant insights from external sources to validate assumptions or suggest modifications to the structure of causal models, enabling a novel fusion of data-driven algorithms and knowledge-based reasoning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.