HCC: Small: Discovering Best Practices for Intelligent Augmented Reality Design in an Ecologically Valid Inspection Setting

Augmented reality (AR) glasses overlay virtual information on the real world and have the potential to provide anytime/anywhere access to relevant information for a myriad of tasks and situations. However, it is challenging to design effective user interfaces for AR glasses, because the best interface depends on multiple dynamic aspects of the context of use. In an industrial inspection scenario, AR glasses can display information that helps the inspector know what to do next, but the most effective placement of this information depends on the objects in the environment, and the most effective level of detail depends on the expertise of the inspector. This project investigates how AR glasses can use artificial intelligence to automatically adapt their interfaces based on contextual information that is sensed in real time. These “intelligent AR” systems will present the right information, at the right time, in the right way. Through a series of studies, both in the lab and in a real manufacturing environment, the project will develop a set of best practices for intelligent AR that will help designers understand when, how, and where to display augmented information based on the context. The work is aimed directly at facilitating economic and safety benefits by improving future inspection and maintenance work through the application of future intelligent AR technology. It is also expected that the results of this project in industrial inspection and maintenance will inform similar applications in other domains that share some of the characteristics of the manufacturing industry, such as dynamic nature, complexity, and the need for workers to have multiple skills. Such domains include construction, military, and healthcare. In addition, the project will provide opportunities for students at various levels to participate in cutting-edge AR research and will include outreach activities that generate excitement about human-centered computing research to young audiences. The technical work in this project makes progress towards the intelligent AR vision by designing, prototyping, and evaluating its use in a specific domain: equipment inspections in a complex manufacturing environment. The project leverages an industry partnership to provide real-world requirements and ecologically valid evaluations for intelligent AR prototypes. The prototypes will go beyond simple context-awareness, which is based on hand-crafted rules applied in specific scenarios. Instead, they leverage machine learning models to prescribe what information is needed, when to present it, and where it should be placed in scenarios never encountered before. A series of controlled experiments will provide focus on three critical research questions: what level of detail to present based on inspector expertise, when to present information based on inferred task progress, and where to present information based on an understanding of the surrounding environment. The results will identify how users prefer to work with intelligent AR, examining issues related to trust, preferences in intelligent automation, and to what degree users wish to control what, when and where information is presented. The project will conclude with a field test of a functioning intelligent AR prototype in a manufacturing environment in order to assess the impact of this approach on real-world inspection tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.