Taming sequential decision-making with reinforcement learning: non-stationarity, heterogeneity, and online/offline comparison

Many decision-making tasks in healthcare, business, and economics can be naturally framed as online sequential decision-making problems, where decisions are made and outcomes are observed iteratively to achieve long-term objectives. Reinforcement learning (RL) offers a powerful framework and has achieved significant success in engineering domains, including robotics and gaming. However, human-centered tasks — such as those in healthcare and business — pose substantial new challenges for RL. These high-stakes tasks are more complex (e.g., non-stationary environments, heterogeneity across objects, and tension between leveraging historical data and the need to perform well in an interactive online setting) and impose more requirements (e.g., interpretability, performance guarantees, and computational efficiency). This project will address these challenges by conceptualizing and gaining insight into the aforementioned complications and by developing well-rounded methodologies that can effectively handle them and meet all requirements. The research outcomes will be broadly applicable to diverse fields, including but not limited to healthcare (e.g., patient treatment, mobile health), business (e.g., operations management, marketing, financial strategies), and economics (e.g., public policy). This project also integrates research and education by providing research training opportunities for students and incorporating the findings into course materials. In more detail, the project will focus on three interrelated tasks: (1) investigating non-stationarity in a principled way and developing methods that are adaptive and robust to it; (2) developing personalized RL models and methods to address heterogeneity and studying its influence and implications; (3) systematically comparing online vs. offline RL through establishing principled criteria and gaining insights to guide algorithm development and evaluation. Individual tasks also include computational components that examine the trade-off between computational cost and statistical accuracy. Collectively, these components provide a well-rounded solution to online sequential decision-making problems in business and healthcare that face multifaceted challenges. This project is interdisciplinary and will leverage not only techniques in reinforcement learning and general machine learning but also ideas and tools from diverse technical subfields (e.g., nonparametric statistics, high-dimensional statistics, optimization, and applied mathematics), as well as domain expertise in business and healthcare. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.