Using Generative AI to Create Illustrative Diagrams to Support Mathematical Problem-Solving

Finding ways to make school math instruction more effective and relevant is important for supporting student pathways in STEM (Science, Technology, Engineering, and Mathematics). Mathematical tasks that describe real life situations represent an area of struggle for many K-12 students. This project uses a novel approach where free, high-quality illustrative diagrams are automatically generated for math problems using recent advances in generative AI. Illustrative diagrams are images that are visually compelling and artistic, while also containing mathematical information such as accurate measurements of geometric figures or the precise number of objects in a collection. Such images have typically been costly and time-consuming to produce; however, in this project, they will be created in a free and scalable way using cutting-edge AI approaches. The research will involve interviewing math teachers and Open Educational Resource developers (who produce free curriculum materials) about their needs and piloting methods for developing AI-enhanced illustrative diagrams in collaboration with these stakeholders. An experiment will then be conducted using the ASSISTments online homework platform to examine the effect of AI-generated illustrative diagrams on middle school students' mathematical learning. Using effective math visuals during instruction is an important way to impact students' interest and performance. The technical approach used in this project involves ControlNets, which allow for rich illustrative features to be layered over mathematical diagrams that are rendered by Generative AI from natural language prompts. ControlNets can generate images that would be impossible to create through traditional prompting, and learning more about how to apply them in contexts where image composition and spatial positioning and measurements need to be precise will have far-reaching implications across many use cases in image generation. The educational research questions are as follows: (1) What needs do teachers and open education resource developers express related to mathematics visuals in their curriculum materials? (2) What practices do teachers and developers engage in when generating their own math visuals, with and without AI? (3) How does the presence of AI-generated illustrative diagrams affect middle school students' interest, performance, and learning? This project will advance computer science, as ControlNets are a new approach and there is limited research on how they can be leveraged. The computer science research questions are: (1) Which types of ControlNet pre-processing models work best for different image types? (2) How can ControlNets be implemented to preserve key properties of mathematical diagrams? (3) How can the process for using ControlNets to generate illustrative diagrams be optimized, and what is the role of autonomous AI agents? This project is funded by the Research on Innovative Technologies for Enhanced Learning (RITEL) program that supports early-stage exploratory research in emerging technologies for teaching and learning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.