Gaze-contingent AI-enabled Low Vision Assistive Technology (GALVAT)

Smart assistive technology (SAT) for individuals with severe vision loss is becoming affordable, thanks to miniaturization of display hardware (head-mounted or phone screen) and more powerful AI software, driven by demand in the consumer market. Unfortunately, these developments do not take into account visually impaired users’ perspective on: 1) ergonomics, i.e., non-visual access to all device functions; and 2) relevance, i.e., functionality that meets their needs and enhances independence. The most often heard complaint is that the device or app overwhelms the user with information, so pertinent information is too hard to find. In this application, four organizations — the Low Vision divisions at Johns Hopkins (JHU) and the Chicago Lighthouse (CLH), Applied Universal Dynamics Corp (AUD) and the Pritzker Institute at the Illinois Institute of Technology (IIT) — are joining forces to address this problem. We are uniquely positioned to do so, through our combined expertise in smart systems design and integration (AUD), clinical and rehabilitative knowledge of the user population (JHU and CLH), and the opportunity to extend the user base to visual prosthesis users (IIT). With the users’ lived experience in mind, we propose to reduce the information overload by defining a gaze-contingent region of interest (RoI) specific to each user’s gaze and remaining visual field, so the AI- supported system will operate on targets within this RoI. In the R61 phase of the project, we will use a head- mounted display (HMD) with built-in eyetracking as the platform; in the R33 phase, a glasses frame with built-in camera and eyetracking will be used, with a smart phone or belt pack computer running the AI engine. The R61 phase will encompass 3 Aims: 1) Formulate user requirements by surveying SAT users (interviews and discussions), and recruit an expert user panel for ongoing feedback; 2) Design and build an HMD-based prototype (H1) for clinical tests, with a software upgrade (H2) based on initial outcomes and feedback; 3) Conduct user training and collect ongoing feedback and performance data for evaluation and re- design. The R33 phase will create the glasses-based system and enhance processing power and customization, through 3 additional Aims: 4) Integrate the glasses frame camera and eyetracker with a speech- enabled belt pack computer (G1), with an upgrade in Year 4 (G2) suitable for take-home use; 5) Enhance user- specific solutions by adding individualized and task-specific RoI adjustments, and add search and hazard warning functions for users with narrow fields; 6) Conduct user training and collect ongoing feedback and performance data for evaluation and G2 upgrades, followed by a 3-month take-home study with performance tests (pre vs, post). The resulting gaze-contingent AI-supported low vision adaptive technology (GALVAT) will be highly adaptive to users’ individual needs and greatly contribute to their functional visual independence.