Collaborative Research: HCC: Computer Aided Design for Nonvisual Computing

Designing precise 3D mechanical objects is exceptionally difficult on low-resolution displays. This challenge is further compounded when shifting from visual creations to tactile interfaces that utilize dynamic, raised-dot arrays. While such tactile technology offers significant potential for the people with limited vision (i.e., the blind and low vision (BLV) community), there is a critical lack of standardized methods for viewing 3D models at these restricted resolutions. This project aims to develop the first fully accessible 3D modeling environment for BLV users by integrating AI-driven view optimization and verbal descriptions with advanced haptic graphics. By enabling the precise manipulation of complex models, this work facilitates new opportunities for professional participation in fields such as engineering, architecture, and industrial design. This opens a new arena of work to BLV people, who can now specify and understand complex and precise physical and mechanical models required for full participation in their careers. The goal of this project is to establish methods for precise and expressive control over 3D model understanding and design. The approach that will be in use is to develop deep collaborations within the BLV community, including co-design with Blind and Deafblind leaders working towards inclusion in tactile graphics and CAD modeling. First, the project will develop foundational new tactile interaction techniques for precisely understanding 3D models, along with novel techniques for richly summarizing features using textual descriptions. AI will support navigation by recommending optimal viewing angles and summarize changes to a model in text. The study will assess participant ability to understand models and model edits, and to connect them to their 3D printed counterparts. Second, the project will support designing new models through learning about, selecting, and parameterizing features applied to specific edges, vertices, and faces of CAD models. AI support will include language-based search both for features and for applying features to models. Finally, the project will support editing existing models. This will require natural language summaries of geometric changes and debugging that can enhance the model editing experience alongside tactile displays. The project will develop educational materials and test users’ ability to set and achieve editing goals and debug models. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.