EAGER: TaskDCL: Collaborative Research: IMPACT: Interactive Mixed-Reality-Based Platform for AI-Driven Adaptive and Collaborative Task Training Environments

Advanced AI-driven training platforms can revolutionize education, workforce development, and specialized training, such as emergency response preparation, by providing realistic, cost-effective, and interactive environments. Progress in this area has been hindered by the need for more affordable and adaptable platforms capable of creating realistic, real-time, closed-loop environments. This EArly-concept Grants for Exploratory Research (EAGER) project aims to develop research infrastructure to enhance human training through AI-driven task environments integrating humans with virtual scene simulations and multi-modal sensorimotor interactions, which holds significant societal benefits. By overcoming current technological limitations, AI-driven task environments can improve the quality and accessibility of training for various applications. The technology this award aims to develop has the potential to significantly reduce training costs, enhance learning experiences, and better prepare individuals for real-world challenges, ultimately benefiting society as a whole. Additionally, the project will introduce K-12 students to cutting-edge mixed reality and AI technologies, sparking their interest in STEM fields. This research will first develop the platform for multi-modal sensorimotor interactions to ensure the best immersive and smooth interactions between humans and virtual scene simulations. It will then establish theoretical foundations and develop efficient algorithms for a closed-loop AI-driven scene task environment. In particular, the research will encompass three interdependent thrusts: 1)Establishing an edge-assisted mixed-reality infrastructure to provide immersive environments and enable smooth sensorimotor interactions between humans and virtual scene simulations. 2)Creating a wide range of training tasks and realistic sensorimotor interactions using flexible and composable modules. 3)Developing a multi-agent reinforcement learning engine to enable dynamic virtual scene generation and adaptation based on interactions. Collectively, this project will produce an immersive mixed-reality infrastructure that ensures smooth sensorimotor interactions, supports real-time task execution, and allows for the exploration of innovative multi-agent learning algorithms. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.