Theoretical Foundations of Long-Term Reinforcement Learning under Uncertainty

Reinforcement learning has achieved remarkable success in simulated decision-making tasks, but deploying such systems in complex, real-world environments remains a critical challenge due to the mismatch between training simulations and actual deployment conditions. Furthermore, standard learning approaches often prioritize immediate gains, resulting in short-sighted behaviors that perform poorly when systems operate continuously over extended periods. In safety-critical applications such as healthcare, power grid management, and autonomous transportation, this vulnerability and lack of long-term focus can lead to severe system failures. This project advances the foundational science of artificial intelligence by developing decision-making algorithms that specifically optimize for steady-state, long-term performance. By ensuring that autonomous agents remain robust against unexpected environmental changes and maintain reliability over indefinite horizons, the research supports the safe integration of autonomous systems into society. The project also integrates research with education by developing open-source software libraries, creating new curriculum modules and robotics capstone projects, and providing specialized training opportunities for undergraduate and graduate students. The technical goal of this project is to establish the theoretical and algorithmic foundations for robust average-reward reinforcement learning, which optimizes long-term steady-state performance under model uncertainty. The research addresses three primary limitations of current frameworks: a lack of finite-sample complexity guarantees, underdeveloped multi-agent formulations, and restrictive single-recurrent-class assumptions. To overcome these challenges, the project is organized into three central thrusts. First, the principal investigator designs a model-free learning algorithm using a quotient-space fixed-point formulation and establishes minimax lower bounds to characterize intrinsic statistical hardness. Second, the project advances multi-agent systems by proving the existence of robust Nash equilibria and developing convergent equilibrium-finding algorithms based on robust temporal difference residual minimization. Third, the research develops multichain robust optimality equations by resolving non-unique worst-case transition kernels through a lexicographic gain-bias separation. The algorithms and theoretical models are evaluated through mathematical analysis, continuous-state simulations, and practical deployments on electric vehicle mobility simulators and quadruped robotics platforms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.