Multi-modal interface for non-visual navigation

There are over 7 million US citizens and 250 million people world wide who suffer from vision loss. Blind and visually impaired individuals face significant hurdles while navigating through the world. Smartphone apps exist to help guide these individuals, but using a smartphone camera occupies a hand, while the user’s other hand holds a cane, guide dog, or other items. For the app to function, the user must scan a scene while holding their smartphone in their hand, which is not like natural vision where we move our head and eyes to obtain information about our surroundings. In addition, task specific apps that only perform a single function require the user to manage which app is currently active, further decreasing usability. Thus, the smartphone interface limits the usability, effectiveness, and adoption of assistive technology. This project will create hardware and software to increase the functionality, and the impact, of the myriad single function apps currently available. We will build a multimodal user interface embedded in a glasses frame specifically to provide non-visual guidance for navigation. A camera will obtain environmental information for apps that rely on real-time video. The interface will be controlled by a smartphone app that serves as an intelligent assistant for the user, by integrating information from multiple sources and automatically launching single task apps when needed. This system will be hands free and will relieve the user from opening and closing multiple apps while also navigating, thus addressing two deficiencies with current assistive technology. To increase user confidence, our system will provide multi-modal sensory feedback, with information provided via vibrotactile and auditory channels. Our prior work supports the immediate, intuitive nature of vibrotactile cues while verbal descriptions can provide more detailed information. The smartphone will provide necessary resources (computational, sensing, connectivity). The system will include a smartphone for two main reasons. First, smartphones are ubiquitous. They have accessibility modes that help blind users navigate through screens. Second, smartphones will, for the foreseeable future, have the most up-to-date processors and connectivity, since the consumer market is so large. Our approach will move only necessary functions of the phone to the glasses, specifically, the camera, sound display, and tactile display, which will constrain cost, improve aesthetics, and delay obsolescence. The system will be tested in a small, feasibility study to refine the design, followed by a larger multisite trial to demonstrate benefit.