Theoretic and algorithmic foundations for provable safe inverse reinforcement learning

This NSF project will lay the foundation for artificial intelligence (AI) systems that are not only effective, but also trustworthy, transparent, and responsible. It will bring transformative changes to AI methods that learn useful behavior from examples while respecting the safety constraints required in high-stakes settings such as robotics, healthcare, and finance. The intellectual merit of the project includes advancing the theory and algorithms of safe inverse reinforcement learning (IRL), a framework in which an AI system learns both a reward function and underlying constraints from demonstrations so that the resulting policy reproduces safe and effective behavior. The broader impacts of the project include accelerating the deployment of AI systems in human-centered and safety-critical domains with greater confidence and accountability, while also facilitate workforce development at the intersection of AI, applied mathematics, and robotics. This project will develop theoretical and algorithmic foundations for provable safe IRL through a unified bi-level optimization perspective, in which the upper level learns a reward function and the lower level learns the corresponding constraints and policy. This framework addresses two major limitations in inverse constraint learning: (i) the assumption that the demonstrator’s reward function is available; and (ii) the lack of safety guarantees for learned policies. The research has two complementary thrusts. The first thrust studies safe IRL in an offline setting, where all demonstration data are collected before training begins. For this case, we will develop a double-loop learning algorithm in which the inner loop uses dual methods to analytically solve the lower-level nonconvex optimization problem, and the outer loop uses the implicit function theorem to learn the reward function efficiently. The second thrust studies safe IRL in an online setting, where demonstrations arrive sequentially during training. For this case, we will develop a single-loop learning algorithm based on two-timescale stochastic approximation, so that the reward, constraints, and policy are updated only once as each new demonstration arrives. The algorithm design and theoretical analysis are derived from integrating tools from optimization, reinforcement learning, stochastic approximation, and sensitivity analysis. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.