CAREER: Enhancing Adaptability in Multimodal Human-Robot Interaction

As robots become increasingly helpful in factories, hospitals, and homes, they must learn how to interact effectively with people. When people communicate, they use words and body language—like hand movements and eye contact—to share what they mean. In the same way, robots that work closely with people need to understand both verbal words and nonverbal signals. One major challenge in creating such robots is that people’s behavior can change over time. For example, a person may use different gestures (e.g., pointing vs eye gaze) to describe an object at different times. The robot must notice these changes and adjust its responses. Also, if the surroundings change, robots must change how they act to keep being helpful. This project will help robots understand people better by studying both verbal messages and nonverbal cues. It will also improve how robots learn and plan to adjust when people’s behavior changes. The results of this research will impact people and help them to be more accepting of robots. The project will help students in several ways. First, new topics will be added to robotics classes. Second, the research will involve high school and college students. Finally, the team will share robotic advancements with the public through outreach activities. This project will focus on three goals to address challenges in multimodal human-robot interactions. First, it will develop advanced algorithms to understand human intentions. These algorithms will use multimodal data, such as verbal messages, hand gestures, eye gaze, and head movements. Second, the project will develop new learning and planning methods for robots. These methods will help robots plan and act in ways that align with human intentions. Lastly, the research will design adaptive learning techniques. These techniques help robots adapt to changes in human behavior to support long-term interactions. The proposed system will be validated through several comprehensive human-robot interaction studies. This project will move the HRI field forward by showing how we can create effective and adaptable long-term interactions between humans and robots. The results of this research will impact how people perceive and accept robots that will be more prevalent in their lives. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.