RI: Uncertainty Quantification for Robust Decision Making

Modern artificial intelligence (AI) systems are now used to take action, not just to make predictions. Examples include autonomous cars, drones, and coding agents. Acting in uncertain environments requires a theory of uncertainty quantification that can be embedded in decision making pipelines. Current techniques for uncertainty quantification often promise only that they are correct on average. Such guarantees can be misleading in exactly the situations where managing uncertainty matters most, such as in situations in which rare, difficult, costly, or safety-critical decisions are the right ones to make. This project will create new foundations for uncertainty quantification that remain reliable when they are used inside decision-making systems. The resulting methods will help make AI systems more dependable, efficient, and useful in settings where mistakes can be expensive or harmful. The project advances the national interest by promoting progress in science and engineering, supporting trustworthy, empowered AI in high stakes settings, improving tools relevant to health and other critical applications, and training students in methods for building reliable AI systems. Educational materials, software, and workshops developed through the project will be made broadly available. Technically, the project will develop model-free uncertainty quantification methods for prediction sets, with a focus on conformal prediction and related calibration methods. The research will address three limitations of current approaches. First, it will design algorithms with coverage guarantees that hold conditional on the downstream action chosen using a prediction set, including settings where the action depends on the uncertainty estimate itself. Second, it will optimize the usefulness of prediction sets, including their size and their value to risk-aware decision makers, while maintaining valid coverage constraints in both distributional and sequential adversarial settings. Third, it will develop short test-time interaction protocols that allow a predictive model and a domain expert to combine different information sources before settling on a prediction or action. The project will evaluate these methods in compound AI systems, including retrieval augmented question answering, visual reasoning systems, and medical decision support tasks. Expected outcomes include new theory and algorithms, open-source implementations, evaluation benchmarks, and educational materials on uncertainty quantification for robust decision making. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.