HCC: Foundations of Automated Evaluation and Monitoring of AR Guidance in Pervasive AR and Human-Robot Collaboration (HRC) Applications

Augmented reality places computer-generated directions, labels, and warnings into a person's view of the real world. These systems are becoming useful for navigation, training, maintenance, health care, warehouse work, and human-robot collaboration. However, augmented reality guidance can be hard to read, appear in the wrong place, provide incorrect instructions, hide important real-world information, or encourage actions that are unsafe, intrusive, or poorly suited to the surrounding environment. A user may not always notice these problems before acting on the guidance. This project will develop new ways to automatically check whether augmented reality guidance is clear, correct, safe, and appropriate in the physical setting where it appears. The work will help make augmented reality systems more trustworthy and easier to use in everyday and safety-sensitive settings, including settings where humans and robots work alongside each other. It will also support education and workforce development by training students in augmented reality, robotics, artificial intelligence, edge computing, and human-centered system design, and by sharing datasets, software, and demonstrations with researchers, industry, and the public. This project will create foundations for automated evaluation and continuous monitoring of augmented reality (AR) guidance, including AR used in human-robot collaboration. The research will focus on five common kinds of guidance failures: poor legibility, inaccurate spatial alignment, incorrect guidance, unsafe guidance, and guidance that is not contextually appropriate. The team will collect and annotate large datasets of AR guidance examples across varied physical environments, lighting conditions, tasks, and robotic collaboration settings. The project will develop evaluation methods that combine AR device data, auxiliary computer vision models, and large multimodal models that can reason about images, text, and scene context. For robotic settings, these methods will also incorporate robot perception, planning, motion, and prediction data to assess whether AR guidance supports safe and effective human-robot coordination. The project will create real-time monitoring platforms that run across AR headsets, edge servers, cloud systems, and, in human-robot collaboration settings, robots. The research will also study how to notify users, adapt AR content, or adjust robot behavior when guidance quality is degraded. Additional work will reduce computation, communication, latency, and cost through data-efficient processing, temporal memory, prediction, and resource-aware system design. The resulting datasets, open-source tools, and evaluation methods will provide a practical foundation for safer and more reliable AR guidance in both pervasive AR applications and AR-mediated human-robot collaboration. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.