CAREER: Urban-Focused Spatial-Temporal Offline Reinforcement Learning

Urban applications such as transportation, logistics, ride-sharing, food delivery, and emergency response rely on millions of decisions every day. Improving these decisions can enhance efficiency, sustainability, and quality of life. Offline reinforcement learning, which learns from historical data rather than real-time trial-and-error interactions, has emerged as a promising paradigm for optimizing decision-making in such complex urban systems. However, applying offline reinforcement learning in urban environments remains challenging. Human-generated urban data are often sparse, heterogeneous, and collected under rapidly changing conditions, making it difficult to learn reliable policies that adapt across varied urban contexts. Moreover, decision patterns vary substantially across space and time, requiring policies to generalize across geographic regions, temporal contexts, tasks, and human behaviors. These challenges are further compounded by complex interactions among decision-makers and the increasing prevalence of human-in-the-loop instructions. This project addresses these challenges by advancing reliable and adaptive decision-making in complex urban environments, enabling robust and human-centered technologies that bridge the gap between advances in artificial intelligence and their practical deployment in cities. The resulting technologies aim to improve urban efficiency, resilience, sustainability, and responsiveness. In addition to these technological benefits, the project integrates research with education and workforce development through curriculum innovation, student training, outreach activities, and collaborations with industry and public-sector partners. The technical goal of this award is to develop a novel Spatial-Temporal Offline Reinforcement Learning framework that enables reliable and adaptive decision-making in complex urban environments. The research is organized around three integrated thrusts. The first thrust develops new methods for learning from human-generated spatial-temporal data by inferring reward signals from unlabeled trajectories, modeling spatial and temporal variations in human behavior, and adapting to different levels of decision-maker expertise. The second thrust advances robust and generalizable offline reinforcement learning by improving policy reliability under distributional shift, enabling multi-task learning from sparse and heterogeneous urban data, and supporting zero-shot adaptation to previously unseen urban scenarios through the integration of large language models and trajectory representations. The third thrust investigates interactive urban decision-making by developing methods for multi-agent coordination, inferring shared objectives from decentralized behaviors, and leveraging large language models to enable instruction-guided coordination from multi-modal urban data. Collectively, these efforts will establish a new scientific foundation for learning reliable, adaptive, and human-centered decision-making policies from human-generated spatial-temporal data in complex urban environments. All resulting models, algorithms, datasets, and evaluation frameworks will be released as open-source resources to accelerate research and innovation in urban intelligence and offline reinforcement learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.