Pocket TA: A Multimodal AI Platform for Personalized, On-Demand Scaffolding to Advance Engineering Education and U.S. Workforce Competitiveness in the AI Era

This project investigates a personalized tutoring assistant (TA) for electrical and computer engineering that is a smartphone-based multimodal platform designed to explore fundamental questions about how perception, reasoning, and scaffolding can be integrated to support engineering education. A systematic investigation is conducted to explore how multimodal artificial intelligence (AI) systems can be designed to support authentic engineering learning, and how they affect student cognition and problem-solving development. This research enhances undergraduate engineering education, contributes open research infrastructure, and establishes transferable STEM approaches. Students will gain curriculum-grounded, on-demand AI support when TA assistance is unavailable. The primary objective is to advance both AI and learning sciences through convergent research that investigates interconnected technical domains and learning science domains by deployment in authentic educational contexts. This project advances fundamental research in the following computer science and learning science domains: graph-based circuit reconstruction, multimodal sensor fusion, curriculum grounded large language model (LLM) orchestration, and artificial intelligence (AI) -mediated metacognitive development. This project investigates questions at the intersection of computer and learning sciences through a platform exploring how multimodal perception, curriculum-grounded reasoning, and Socratic dialogue advance both fields. The research asks: (1) Can computer-vision algorithms reconstruct circuit topology from smartphone images? (2) Can multimodal sensor fusion enable real-time diagnosis of circuit states and faults? (3) How does AI tutoring grounded in official course materials affect metacognition and conceptual reasoning? Project outputs include novel algorithms, annotated multimodal datasets, theoretical models of AI-mediated learning, and design principles for curriculum-grounded educational AI. This research advances scientific knowledge and delivers societal benefit by enhancing undergraduate engineering education, contributing open research infrastructure, and establishing transferable STEM approaches. The AI objectives address fundamental perception and reasoning challenges that extend beyond current capabilities. The first objective is to develop and validate novel graph neural network architectures that can reconstruct electrical circuit topology from smartphone photographs despite visual ambiguity, occlusion, and varying imaging conditions. The second objective is to create and evaluate multimodal sensor fusion methods that enable real-time diagnosis of dynamic circuit behaviors using only commodity smartphone sensors, addressing temporal alignment challenges between audio sampled at forty-eight kilohertz and video captured at thirty frames per second while maintaining inference speeds suitable for mobile deployment. The third objective is to investigate large language model orchestration techniques that ensure factual consistency between AI-generated explanations and authoritative course materials retrieved in real-time from learning management systems, while simultaneously maintaining pedagogically appropriate Socratic dialogue structures that require student reasoning before providing hints. The fourth objective is to examine how Socratic AI tutoring that requires evidence-based reasoning before providing hints affects metacognitive development, help-seeking patterns, and the evolution of troubleshooting strategies as students move from novice trial-and-error approaches to expert systematic debugging across multiple courses spanning sophomore through junior year. The fifth objective is to investigate effects explicitly through analyses and purposive sampling, and identify design features that promote engagement rather than presuming prior knowledge or technological fluency. This project is funded by the Research on Innovative Technologies for Enhanced Learning (RITEL) program that supports early-stage exploratory research in emerging technologies for teaching and learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.