STTR Phase II - Development of a visual-to-tactile conversion system for automating the tactile graphic generation process

PROJECT SUMMARY/ABSTRACT The goal of this Phase II STTR project is to develop an intuitive AI-powered system, called Midlinatm, which will automate the conversion/generation of accessible STEM materials from visual formats and flexibly deliver the accessible content across multiple hardware and software platforms to support blind and visually impaired (BVI) learners. Timely access to STEM materials is one of the biggest challenges for BVI learners, limiting their productivity and success. While access to textual information has largely been solved via screen reading software (e.g., JAWS or VoiceOver), very little progress has been made in making non-textual (i.e., graphical) information accessible for BVI people. With STEM curricula being dominated by non-textual information (e.g., diagrams, graphs, charts, images, mathematical notations), the visual nature of this critical information prevents more than 660,000 school-aged (ages 4-21) BVI learners from pursuing STEM curricula. Current technologies/devices address the issue via human-driven approaches using specialized single purpose hardware that suffer from several shortcomings including high cost, limited portability, lack of multi-purpose, and inability to present information in a real-time context. Importantly, a common underlying problem across all extant approaches is that they require intensive human efforts for producing or authoring tactile (and/or multimodal) graphics. These approaches lead to high production costs and significant delays from the time the accessible materials are needed to when they are actually delivered; ultimately adversely impacting BVI individuals’ learning and advancement in K-12 schools, colleges, and performance in workplace settings. To address this long-standing problem, in Phase I, UNAR Labs developed a prototype version of Midlina’s framework using Math worksheets as an example and demonstrated its practical utility by implementing it on a prototype web application, called Kanak. In this Phase II effort, UNAR Labs will build on the success of Phase I by extending the capability of Midlina’s framework and establishing it as a commercially viable platform technology for generating/converting STEM materials into accessible formats. We will also develop and deploy two Midlina-powered applications, (1) Kanak –a web application aimed at aiding teachers and TVIs with real-time accessible document creation, and (2) MathAlly – an IOS app that helps BVI learners access accessible math materials in real-time. The team is highly experienced in developing assistive technology solutions. Successful completion of the project will provide a robust system for increasing the participation of BVI individuals in STEM programs and will make educational institutions more inclusive and equitable.