Methods for Analysis of Nonlinear Filter with Applications to Reinforcement Learning

This award supports foundational research that contributes new knowledge related to control theory, thereby promoting both the progress of science and advancing national prosperity. Specifically, the project is concerned with the duality theory between optimal control and nonlinear filtering with applications in reinforcement learning (RL). On one hand, duality has been a central topic in control and estimation theory for years, yet there is still a lack of deep understanding beyond linear quadratic Gaussian systems. On the other hand, a key limitation of the state-of-the-art RL algorithms is that the theoretical foundations are less than well-understood and practical implementations can require massive amounts of data and computational resources. This project will address the above critical gaps by providing needed knowledge for the analysis of the nonlinear filter and development of efficient types of RL algorithms. RL algorithms are increasingly being adopted for a number of applications in engineering and sciences such as self-driving autonomous cars and generative algorithms. The research plan is closely integrated with an educational plan whose goals include developing and maintaining a strong undergraduate involvement in research and fostering a symbiotic relationship between research and entrepreneurship. This research aims to make fundamental contributions to nonlinear filtering and optimal control. It will achieve this goal using two parts. Part I concerns the development of methods for stochastic stability theory for the nonlinear filter and part II concerns applications of such methods to obtain a new class of RL algorithms. The specific research tasks are to relate the properties of a hidden Markov model (HMM), specifically ergodicity and detectability, to the asymptotic stability of the nonlinear filter. Several analytical techniques and extensions are planned including a study of Hamilton’s optimality equations, the development of dissipativity theory for HMM, and an analysis of the singular deterministic limit. The theoretical research is closely integrated with the work on algorithms and computational methods for RL. These algorithms are based on interacting particle systems, namely, dual ensemble Kalman and feedback particle filters, for solving the linear and nonlinear optimal control problems, respectively. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.