HCC: AI-driven Platform for Training Social Skills by Participation in Spontaneous, Adaptive, Multimodal Interaction Scenarios

Social skills such as the abilities to navigate complex interpersonal situations, infer others’ intentions, and respond in real time, are foundational to successful interactions. This project develops an AI-driven training platform in which people can participate in open-ended situations with virtual agents to learn and practice social skills. The virtual agents combine language with multimodal capabilities such as gesture, gaze, posture, and facial expression, to engage with people in complex face-to-face interactions. A central scientific focus is Theory of Mind (ToM), the human ability to reason about others’ beliefs and intentions. The platform is explicitly designed to train and evaluate people's ToM skills, as well as assess the uncertainties in their beliefs. The approach provides adaptive training for social skills across diverse domains such as education, healthcare, security, intercultural communication, and mental health. A multi-agent large-language model (LLM) framework is utilized consisting of: 1) a “designer” LLM that collaborates with learners to create personalized scenarios, 2) a “director” LLM that dynamically sets up and influences scenarios to enhance flexibility, engagement, and learning, 3) multimodal virtual human LLMs that provide rich verbal and nonverbal behaviors that convey underlying beliefs, attitudes, uncertainties, and social cues, and 4) a “learning and alignment” LLM that combines human feedback, preference modeling, and continual evaluation to iteratively refine agent behaviors, improving adaptability, and realism across scenarios. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.