Sensible AI: Integrating Multimodal AI and Mixed Reality for Context-Aware Real-Time Laboratory Learning in Microelectronics

The United States’ leadership in artificial intelligence (AI) and national security depends on a robust semiconductor manufacturing ecosystem supported by a highly skilled technical workforce capable of advanced AI chip fabrication and security analysis. Physical inspection and security assessment of chips demand critical decision-making in high-stakes laboratory environments, where engineers operate multimillion-dollar equipment and perform tasks requiring nanometer-level precision. The conventional approach to traditional laboratory training presents two critical limitations: it deprives learners of immediate, context-specific feedback when corrective action would be most effective, and it reduces training to procedural steps that fail to convey conceptual understanding or the underlying principles of the process. To overcome these limitations, the project introduces a predictive cognitive–behavioral modeling framework that fuses temporal attention mechanisms with probabilistic reasoning to interpret expert–novice divergences during live chip inspection tasks by tailoring scaffolded support based on each learner’s cognitive state and level of expertise. The research objectives of this project are to: characterize expert decision-making strategies through multimodal datasets, develop a multimodal AI-mixed reality framework that dynamically interprets learner cues to generate adaptive instructional feedback, and evaluate the impact of Sensible AI on learners’ decision-making, transfer of learning, and professional readiness in semiconductor education. Guided by this interdisciplinary foundation, the project explores the following overarching research questions, 1) How do expert laboratory engineers engage in microelectronics physical inspection tasks, and what decision-making processes characterize expert performance? 2) How can multimodal data be integrated into model expert decision-making during microelectronics physical inspection? 3) How can a multimodal AI system dynamically interpret learners’ real-time cues to infer their evolving knowledge states and instructional needs? 4) How can context-aware mechanisms be designed to dynamically align instructional scaffolding with learners’ developmental stages of expertise. 5) How do Sensible AI’s adaptive scaffolding strategies influence learners with different levels of prior knowledge in terms of their conceptual understanding, procedural accuracy, and decision-making during laboratory learning? 6) To what extent does learning through the Sensible AI framework transfer to novel laboratory tasks and contribute to learners’ career readiness in the field of microelectronics? This research contributes generalizable principles for AI-enabled learning in complex physical domains, establishing new frontiers at the intersection of learning analytics, human–AI interaction, and semiconductor training. This project is funded by the Research on Innovative Technologies for Enhanced Learning (RITEL) program that supports early-stage exploratory research in emerging technologies for teaching and learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.