I-Corps: Augmented Reality-Powered 3D Interactive Instructions

The broader impact/commercial potential of this I-Corps project is the development of Augmented Reality (AR)-powered 3D interactive instruction technologies for broad applications in block toys, furniture, manufacturing, building construction, and educational services industries. The products in these industries are becoming increasingly complex and confusing instruction manuals can lead to poor construction or missed educational opportunities for users. By providing the customers with advanced AR-based instructions, the proposed technology is expected to help improve the assembly processes and increase the educational information. The broader impacts of the technology may impact education, design, gaming, medicine, and other industries. This I-Corps project is based on the development of an Augmented Reality (AR)-based technology for the assembly, manufacturing, construction, and education industries. Applications of the proposed technology have been prototyped and evaluated. Using the proposed technology, physical assembly may be guided by virtual parts. In this platform, the technology of realistic object occlusion reveals the true spatial relationships between physical and virtual parts and between users’ real hands and virtual parts, in the step-by-step assembly process. The proposed virtual-physical model alignment is highly accurate using various AR registration methods. The integration of these features makes AR instructions possible for large, small, or tiny parts assembly, validated through working prototypes for assembling artifacts such as block toys and furniture, and through quantitative measures of the accuracies of registration and occlusions. In addition, a heuristic evaluation of features of the proposed technology has led to findings that the method could advance AR instructions in terms of enhancing part visibility, the match between mental models and visualization, and the alignment of physical and virtual parts in perspective views and spatial transformations. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.