Career: Example-Enhanced Intelligent Tutoring

The rapidly evolving workplace landscape calls for the development of scalable upskilling and reskilling programs for maintaining a competent and competitive workforce. According to the World Economic Forum’s 2023 future of jobs report, six in ten workers will require training before 2027, but about half of the workforce does not have access to adequate training opportunities today. From what we know about human learning, it is clear that deliberate practice, appropriate scaffolding, and timely feedback are needed for learning to be effective. Practical implementation of such features requires substantial investment from domain experts or instructors, which makes it difficult to provide these opportunities at scale. The overarching goal of this project is to make expertise sharing more efficient through helping experts create example-based intelligent tutors. The research team will partner with other universities, local community colleges, and public schools to test the developed tools, which will then be made publicly available. Through this work, the project will make it easier to develop effective training programs that scale well to millions of workers, improving both their own opportunities and the U.S. economy as a whole. This project focuses on the teaching and learning of complex problem-solving tasks, e.g. “Identify a recent economic phenomenon that involves market failure, explain which type of market failure it is, and propose a solution.” To capture the expertise behind such tasks, the project introduces a “Checklist with Examples” approach, which outlines the key criteria a solution must meet. Four main activities guide this project. Thrust 1 involves creating example-based intelligent tutors for task domains that already have detailed checklists. This will produce human-AI collaborative techniques, and a platform (“Exemplify”) for instructors to gather examples that meet specific criteria. Instructors can find these examples from prior students’ homework submissions using retrieval-augmented generation (RAG)-based methods or generate them directly with Large Language Models (LLM). Exemplify also helps instructors create scaffolding exercises such as multiple-choice questions or example-annotation tasks with automated feedback. Thrust 2 involves performing randomized controlled classroom experiments to evaluate the resulting example-based intelligent tutors. Thrust 3 involves developing a feedback engine using the checklists and the examples collected earlier. The engine provides students with automated formative feedback as they work on open-ended problem-solving tasks. Thrust 4 develops a low-cost proxy method for cognitive task analysis by having LLM agents simulate novice learners and generate solutions. Experts review these solutions and their feedback is summarized into a checklist, making it easier to capture expertise in any domain. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.