Risk-Sensitive Reinforcement Learning via Coherent Risk Measures: Framework and Algorithms

Reinforcement learning (RL) is an area of machine learning where agents learn from interacting with environment to determine actions. RL tools have been widely used in many different engineering systems such as power grid, wireless communications, and autonomous driving etc. A common goal in these decision-making tasks is to determine an optimal policy that minimizes the expected total discounted cost, which is also named risk-neutral approach. Although the risk-neutral approach is quite popular, it does not take the tail of the distributions of the cost into consideration. In the tail of the distribution, the cost may be prohibitively high, even though the probability of happening is low. In safety-critical applications, it is important to consider these rare but consequential events. Given the potential drawbacks of risk-neutral approach in safety critical applications, there is a pressing need to systematically study the risk-sensitive approach, in which one designs decision policies that take the risk into consideration. The goal of this project is to develop a unified framework for the design of efficient algorithms for risk-sensitive RL by systematically employing a class of risk measure named coherent risk measures and develop efficient algorithms that could be implemented in engineering systems. Even though a multitude of risk measures have been extensively studied in the literature and successfully applied to RL, existing work face the following challenges: 1) Most of these applied risk measures are not coherent; 2) While there is an increasing demand for a broader range of choices in risk measures to better align with individual risk preferences in complex scenarios, there is a lack of unified framework that enables efficient design of risk-sensitive RL algorithms tailoring to users’ different choices of risk measures suitable for their applications; and 3) There may be model uncertainties and model shifting in practical applications, but existing risk-sensitive RL algorithms are not robust to such scenarios. To address these challenges, this project is focusing on two interconnected thrusts. In the first thrust, the research team aims to develop a unified framework for the efficient design of risk-sensitive RL using coherent measures tailoring to users’ different choices of risk measures suitable for their applications. In the second thrust, based on the developed framework, the team aims to design robust risk sensitive RL algorithms that are robust to model uncertainties and model misspecifications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.