Collaborative Research: CNS Core: Small: SmartSight: an AI-Based Computing Platform to Assist Blind and Visually Impaired People

This project will design SmartSight, a platform that combines the computing ability of the device-to-cloud continuum with Artificial Intelligence (AI) methods to enable ambient perception for blind and visually impaired people (VIP), thereby improving their quality of life and social inclusion. SmartSight operates based on smart glasses and provides real-time perception for its user by efficiently deploying machine learning and approximate computing methods across a continuum of tiers—from local tiers (device and edge) to remote tiers (fog and cloud platforms). SmartSight uses federated learning to take advantage of collective observations of users and improve its perception accuracy. The platform is robust against failure in its communication with remote tiers and is able to provide its basic functionality using only local tiers. For that purpose, SmartSight dynamically adjusts approximate computing knobs and allocates critical tasks to resources, only if they can produce timely results. A prototype of SmartSight will be developed and tested with blind and VIP to assess its usability, robustness, convenience, and discreteness. SmartSight enables “AI+Systems for social good.” The project results will impact not only the forefront of scientific discovery, but also society by improving the quality of life and social inclusion of the blind and VIP. Other citizens who suffer from some form of cognitive decline (e.g., Alzheimer's disease) will benefit from SmartSight technology and be able to function as independent members of society. The steps towards addressing these hurdles not only will improve the quality of life for these people but will also bring about a remarkable cost-saving to society as a whole. The potential gain is even greater when considered on a global scale. The intelligent resource allocation and approximate computing methods that will be developed for the SmartSight platform can be adapted to other soft real-time computing systems, and the proposed efficient and scalable approaches for automated parameter tuning and ubiquitous model execution can be employed in other tunable software systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.