Project MultiNav: A gamified multisensory training and navigation system for blind and visually impaired people

Project Summary Although there has been decades of assisƟve technology (AT) development, navigaƟon‐related challenges remain one of the most significant obstacles to independence and overall quality of life for blind and visually impaired (BVI) people. The starƟng point for this research is the view that most navigaƟon challenges experienced by BVI travelers are not due to vision loss but caused by insufficient informaƟon provided by current navigaƟon technologies. NavigaƟon is a conƟnuous endeavor involving route planning, wayfinding, environmental interpretaƟon and monitoring, and real‐Ɵme movement, but exisƟng AT soluƟons address these aspects in a fragmented way that significantly limits effecƟve spaƟal learning and navigaƟon. This fragmented approach focuses on isolated tasks (pre‐journey learning vs. in‐situ navigaƟon), segregated informaƟon (routes vs. maps), specific modaliƟes (language vs. hapƟc), and siloed technology development (digital apps vs. physical tacƟle maps). This two‐phase R61/R33 project aims to break down silos characterizing decades‐long AT development by developing an integrated mulƟsensory navigaƟon soluƟon, called MulƟNav, which converges pre‐journey map learning with in‐situ navigaƟon assistance, wrapped in a learnable and usable smartphone applicaƟon with strong pathways to translaƟon. To do so, MulƟNav advances the state of the art for mulƟsensory access idenƟficaƟon through development of the first user‐centered design guidelines specifying the key informaƟon/modality (I/M) pairings needed to support the highest probability navigaƟon tasks across the complete trip. UƟlizing these I/M pairings, the MulƟNav app's mulƟsensory mapping system provides users with route and environmental overviews for use in both pre‐journey exploraƟon and during real‐Ɵme travel, enabling BVI people to obtain a level of environmental access, navigaƟonal assistance, and spaƟal learning capabiliƟes that are simply not possible from current navigaƟon AT soluƟons. A significant innovaƟon of MulƟNav's approach is that route guidance and map overview informaƟon are coupled with a unique exploraƟon modes that enables BVI users to learn the real‐world environmental features necessary for safe street crossing (e.g., intersecƟon geometry, presence of medians, locaƟons of accessible pedestrian signals), as well as a traffic view mode that provides mulƟsensory Ɵme‐of‐day traffic data to enable users to hear and feel how a crossing will be navigated in advance. These features are introduced to users in a gamified learning mode such that the user interface can be learned, understood, and uƟlized long‐term via engaging navigaƟon scenarios. As a result, MulƟNav's user‐driven inclusive design approach is expected to solve high‐probability navigaƟon challenges with tractable and measurable outcomes that will significantly improve BVI mobility, independence, and overall quality of life as a sustainable plaƞorm with widespread adopƟon.