CAREER: Making Digital Imagery Accessible to Blind and Low-Vision Users via Audiohaptic Dioramas

Blind and low-vision people are largely left out of visual experiences on computers, affecting many aspects of daily life such as navigating, shopping, reading the news, and staying in touch with friends. Computers currently convey images to blind users via simple text descriptions, which does not allow users to form their own interpretations. This project seeks to enhance the quality of life for blind people by introducing the concept of audiohaptic dioramas, which allows users to learn what things look like in a way similar to real life: by exploring objects' three-dimensional structure via touch, gleaning physical details such as depth, shape, and material. Audiohaptic dioramas will work with everyday touchscreens, allowing blind users to touch or point at an image and sense its physical details via intuitive audio and haptic cues. This technology will give blind people greater access to images, films, video games, virtual reality, and the Web. The investigator will partner with leading technology companies to make audiohaptic dioramas available on mainstream consumer apps. The project's educational component will make K-12 textbooks accessible to blind students by supplementing printed images with audiohaptic dioramas. It will also train blind students to become technology designers, a field hard to enter without sight, catalyzing future efforts to make technology more inclusive. This research program advances a decades-long problem in accessibility research: understanding how to make images equivalently accessible to blind users. Through the concept of audiohaptic dioramas, this research will contribute the first techniques for conveying physical details of three-dimensional scenes via sound and haptics in a way that is compatible with everyday touchscreen devices. The research program will follow a user-centered design and evaluation process to ensure that blind users’ needs are met, involving blind community and lab members throughout and partnering with blind-serving organizations. This project will lead to several scientific contributions: (1) a framework for conveying still images’ depth, shape, material, and color intuitively via sound and haptics on a per-pixel basis as users explore them on a touchscreen; (2) techniques for enabling blind people to independently perform everyday visual tasks such as navigation, finding a building entrance, or finding a specific color shirt using their smartphone camera or smartglasses; and (3) techniques for enabling blind people to visualize interactive virtual environments from video games and virtual reality. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.