Collaborative Research: RI: Small: Adapting Foundation Models for Multimodal Sequential Decision Making: New Focuses, Formalisms, and Methods

Artificial intelligence has achieved remarkable success in recent years, largely driven by advancements in foundation models, which leverage complex neural networks trained on vast amounts of data in order to perform a variety of tasks, such as question answering, text summarization, and image generation. This project seeks to extend the success of foundation models to sequential decision-making, where an agent--a programmable entity---interacts with an environment, seeking to accomplish a task by taking a series of actions over time, with each action influenced by the outcomes of previous actions. Sequential decision-making commonly arises in situations characterized by uncertainty, limited resources, or dynamic conditions, where each decision can have an impact on future actions. The objective is to select a sequence of actions that maximizes profits, rewards, utilities, or some other well-defined objective. Adapting foundation models for sequential decision-making is challenging, because high-quality data is often lacking and it requires recognizing task-specific structures and optimizing long-term objectives, where minor differences can drastically change optimal solutions. This project will develop novel methods for overcoming these challenges to significantly increase the applicability of foundation models for a wide range of sequential decision-making applications, such as smart manufacturing, multi-agent systems, and human-machine interaction. This project will develop novel techniques and methods to effectively adapt foundation models to multimodal sequential decision-making. The proposed research will be conducted and evaluated on three thrusts with progressively increasing problem complexity. Thrust 1 studies sequential decision-making problems in textual modalities where the decision-maker only needs to look one step into the future when evaluating the consequences of a proposed action, referred to as contextual bandits. The investigators will develop new techniques such as reward-aware text summarization and mixing foundation model-based and online-learned decision rules that leverage foundation models to warm-start the agent while avoiding being locked into pretrained parameters to improve the performance in the long run. Thrust 2 studies sequential decision-making problems that involve long decision horizons (the full reinforcement-learning problem) and are multimodal. The investigators will develop additional techniques that leverage foundation models for multimodal and hierarchical reinforcement learning. Thrust 3 extends the techniques to the cooperative multi-agent setting, where the foundation models are leveraged to facilitate both centralized and decentralized inter-agent communication, which is crucial for multi-agent coordination. In and outside the classroom, this project will conduct a series of educational and outreach activities, including development of course materials related to foundation models and sequential decision-making, undergraduate research mentoring, and public outreach in local communities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.