I-Corps: Translation Potential of an Artificial Intelligence (AI)-Powered Reading Application

This I-Corps project is based on the development of a software platform that converts paperback books into digital books for people with challenges reading print. Currently, 63% of all 4th graders and 91% of disabled 4th graders in the U.S. do not meet basic literacy proficiency standards. This problem, referred to as the “literacy crisis,” costs the country an estimated $2.2 trillion annually. The underlying cause of this problem is that millions of students with print disabilities, such as dyslexia and low vision, cannot perceive the content of roughly 90% of all books, including paper books from the library as well as digital books. This technology makes books perceivable so students can learn to read. The software can extract text, generate image descriptions, and make book contents accessible through seeing, hearing, and touching. This may help to improve literacy proficiency and mitigate the impact of print disabilities for millions of readers. This I-Corps project utilizes experiential learning coupled with first-hand investigation of the industry ecosystem to assess the translation potential of an automated artificial intelligence (AI)-based software algorithm designed for multi-modal document accessibility. The software architecture leverages computer vision algorithms to detect, sequence, and extract text and graphical components from digitized book layouts. This technology pairs structured, hierarchical image descriptions with page-based visual question answering. Additionally, unlike traditional or assistive reading technologies that deliver visual-only, audio-only or text-only outputs, this approach provides multi-modal and on-demand interaction with distinct textual and visual subregions of print media. This may allow people with print disabilities to individually or collaboratively perceive and interact with print media. The technology is based on empirical research investigating automated layout segmentation and dynamic content delivery interfaces for non-visual access. Technical results validate that the core software architecture extracts and structures complex document geometry, demonstrating a versatile engineering foundation capable of being scaled to accommodate myriad print media formats and users across a wide spectrum of print disabilities. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.