Collaborative Research: AI Tools for Measuring Teachers’ Professional Noticing of Children’s Mathematics

Professional noticing, or the ability to attend to, interpret, and make decisions based on students’ thinking, is widely recognized as a foundation of effective mathematics teaching. However, evaluating professional noticing currently depends on time-intensive scoring by human experts, which limits how often teachers can receive and act upon feedback. Research has explored teachers’ noticing and how they make sense of children’s mathematical thinking; but to date, measuring it validly and reliably has not been fully explored. This project investigates whether generative artificial intelligence tools can score teachers’ noticing responses accurately and consistently when compared with expert human raters, and whether the feedback these tools generate is useful to teachers. By examining how artificial intelligence can evaluate a complex teaching practice, the project informs national efforts to use emerging technologies responsibly in education. The work supports the preparation of a stronger teacher workforce, expands access to timely feedback for teachers, and generates practical evidence about when automated evaluation can be trusted and when human judgment remains essential. The overarching goal of this project is to explore the degree to which generative artificial intelligence tools can, with validity and reliability, measure teachers’ professional noticing of children’s mathematical thinking and generate feedback that supports teacher learning. This project examines four research questions: (1) whether generative artificial intelligence tools conditioned on rubrics and exemplars can produce scores for teachers’ noticing responses that align with expert raters across the components of professional noticing; (2) how consistent those scores are across noticing assessments of varying format and complexity; (3) which prompt engineering and model training approaches yield the most reliable automated scoring; and (4) how preservice and inservice teachers perceive the usefulness of feedback generated from those scores. The research team will administer three established noticing measures, two video-based instruments, and one vignette-based instrument, to approximately 100 inservice elementary teachers and approximately 50 preservice elementary teachers. Expert raters will score all responses using validated protocols. Then, commercially available generative artificial intelligence systems will score the same responses using prompting procedures developed through structured, iterative testing. Statistical comparisons of human and automated scores will provide evidence of score validity and cross-task reliability, while surveys and semi-structured cognitive interviews will examine how teachers interpret and act on automated feedback reports. The project contributes new knowledge about the measurement of complex, practice-based teaching competencies and about the conditions under which automated scoring can complement expert human judgment. Deliverables include validated and openly shared scoring protocols, teacher-facing feedback report models, implementation guidance for teacher preparation programs, and empirical findings on the responsible use of generative artificial intelligence in educational measurement. This project was funded by the STEM K-12 program, which supports fundamental, applied, and translational research on STEM learning in both formal K-12 education and informal learning environments. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.