CISE MSI: RDP: IIS: HCC: Demonstrating Mixed Reality and Edge Computing in Human-Robot Interaction and Collaboration Considering Human Factors

The need for robust, resilient, and sustainable cyberinfrastructure for human-robot interaction and collaboration has surged, emphasizing the importance of human-centered computing (HCC). However, current methods struggle to address the complex dynamics of humans and robots, particularly when considering human behavior, mutual impacts between humans and robots, physical interactions, user perceptions, and potential associated risks. By leveraging mixed reality, developing new algorithms in edge computing, and partnering with social scientists on psychological and ethical impacts, the demonstration project aims to enable immersive, context-aware experiences that enhance the abilities of humans and robots to perceive, understand, and collaborate more effectively. Through research findings, the team seeks to enhance San José State University’s research capacity across multiple domains human-robot areas and HCC. This project builds research capacity, sustainable for training underrepresented students via multidisciplinary collaboration and partnership with local companies. Key project outcomes include pioneering immersive experiences through integrating MR, haptic feedback mechanisms, and wearables into interactive physical-based HRI environments. Additionally, the project significantly advances robotic cognitive abilities and promotes intuitive, intelligent interactions with humans. This project focuses on three primary research objectives (1) forging immersive and context-aware experiences for human psychological and ethical study. Using MR to simulate an emergency department environment to enhance understanding of ethical considerations and physiological impacts in human-robot contexts; (2) empowering robots with enhanced perception, comprehension, and interaction capabilities with their environment. Multi-perceptual sensorimotor interactions in the MR simulation enhance robots' ability to dynamically sense and respond in simulated emergency department scenarios, improving their healthcare-related task performance; (3) improving seamless communication between humans and robots. An tailored edge computing architecture is designed and implemented to facilitate real-time data processing and communication between humans and robots, optimizing interaction pathways and task distribution to enhance patient care outcomes in critical situations. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.