Collaborative Research: III: Harnessing Bilevel Structures for Strategic Decision Making and Large Language Model Alignment

Modern artificial intelligence and decision-making systems frequently face hierarchical structures where an overarching goal depends on the outcome of a nested, subordinate problem, such as aligning an artificial intelligence policy that relies entirely on an underlying optimization of human preferences. This mathematical structure, known as bilevel optimization, captures these nested challenges but is traditionally slow and difficult to apply to large-scale, real-world data. This project addresses the fundamental challenge of scaling bilevel algorithms so they can be integrated smoothly into modern machine learning systems. By creating faster, more efficient computational tools, the project will enable better decision-making in critical domains. The project will lead to social benefits by applying these algorithms to optimize healthcare interventions for children and patients with diabetes, improve agricultural and manufacturing practices to reduce environmental impacts, and design safer artificial intelligence algorithms for more efficient and reliable large language model alignment. The project also supports the training of graduate and undergraduate students in use-inspired research. The project advances the computational foundations of bilevel optimization to ensure scalability in problem size, sample efficiency, and stability under stochasticity. Existing approaches are either bottlenecked by computationally expensive implicit differentiation or rely on emerging Lagrangian approximations that remain confined to preliminary theoretical analysis without accessible tools for machine learning. To bridge this gap, the team of researchers will develop sample-efficient, fully first-order bilevel algorithms and tools that seamlessly integrate into modern machine learning pipelines, accommodate stochastic objectives, and scale to massive problem sizes. The investigators will leverage these new first-order algorithms to overcome existing computational bottlenecks in systems inherently governed by bilevel interactions, unlocking new scalable solutions for multi-agent Stackelberg games with multiple followers, decision-making under uncertainty, and model-based reinforcement learning. Crucially, the project introduces the novel formulation of treating human-artificial intelligence interaction as a foundational bilevel structure to fundamentally improve the alignment of large language models. By mathematically coupling reward modeling and policy optimization instead of treating them as disconnected stages, the research activities address major alignment challenges. These activities include uncertainty-based data selection, integrated reward-policy learning to reduce distribution shift, and online exploration techniques for reinforcement learning from human feedback. The potential contributions are practical algorithms, reproducible benchmarks, and an open-source library that bridges optimization theory and large-scale applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.