Open-Source Artificial Intelligence Methodology for Generating and Validating K–12 STEM Disposition Measures

Reliable and valid surveys are essential for understanding students’ confidence, identity, beliefs, and attitudes toward science, technology, engineering, and mathematics (STEM). However, developing and testing high-quality survey items is often slow, expensive, and difficult. This project will develop AI-GENIE-Edu, an open and reproducible system that uses artificial intelligence (AI) and modern measurement methods to help researchers and educators create and evaluate self-report survey items for K-12 STEM education contexts. The project will focus on STEM self-efficacy, STEM self-concept, growth mindset, productive struggle, and attitudes toward artificial intelligence literacy. By improving the quality and accessibility of educational measurement tools, the project will support stronger STEM and STEM education research, more trustworthy evidence, and better-informed educational decisions. The research team will extend the existing AI-GENIE methods to K-12 STEM Education dispositions and examine whether the structure estimated from item text agrees with the structure found in student responses. The project will compare language models, prompting methods, and generation settings, identify redundant and unstable items, and evaluate structural consistency across student groups. Generated items will be tested with high school students and/or first- and second-year undergraduate STEM and STEM education students. The project will also examine whether AI-GENIE-Edu-screened item pools produce more stable measurement results than unscreened artificial intelligence-generated and existing items. Products will include an open-source R package, benchmark datasets, documented prompts and analysis workflows, research publications, tutorials, webinars, and professional-development materials. This project was funded by the STEM K-12 program, which supports fundamental, applied, and translational research on STEM learning in both formal K-12 education and informal learning environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.