CAREER: From sensing to action: Enabling robot speech, gestures, and touch to provide supportive responses for human emotional distress

In the United States, new patients often wait six to eight months to see a mental health provider. For individuals experiencing grief, anxiety, or severe emotional distress, these delays can leave critical gaps in care during moments when support is most needed. Research shows that effective emotional support often combines both verbal communication and physical gestures, such as comforting touch. However, most current technologies provide only text- or voice-based responses and lack the ability to recognize emotional signals or respond through embodied interaction. This CAREER project investigates how robots can recognize signs of emotional distress and provide supportive responses through speech, gestures, and gentle touch from miniature humanoid and industrial robots in social contexts. By enabling robots to perceive and respond to emotional cues in real time, this research aims to create technologies that can help individuals regulate emotions and feel socially supported. The project also integrates research and education by creating interdisciplinary robotics programs that engage students across engineering, the arts, humanities, and social sciences, while expanding outreach through K-12 workshops, internships, and public demonstrations. This project develops a real-time, closed-loop human-robot interaction framework that enables robots to sense, interpret, and respond to human emotional states during social interaction. First, the research will develop interpretable computational models that infer emotional states from multimodal off-body signals, including heart rate, respiratory rate, speech prosody, facial thermal patterns, and posture. Second, the project will design supportive robot behaviors by developing therapist-guided verbal responses and constructing a gesture library derived from annotated videos and motion capture data of natural human comforting behaviors. Third, these sensing and behavior modules will be integrated into an adaptive robotic system that dynamically selects and modulates tactile and verbal responses based on a user's evolving emotional state and social context. Controlled user studies will evaluate the system by measuring perceived empathy, appropriateness of responses, and emotional outcomes. The resulting framework will establish new principles for emotionally intelligent human-robot interaction and provide open datasets and tools that enable future research in robotics, human-centered AI, mental health support, and assistive technologies. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.