Harvesting Actionable Results for Learning and Instruction: A Novel Mixed Methods Approach to Extracting and Validating Information from Diagnostic Assessment

Cognitive diagnostic models (CDMs) are an area of psychometric research that has seen substantial growth in the past decade. These assessment tools have received more attention because a simple overall test score does not serve teaching goals as a richer evaluation of the student's skills is needed to support tailored instruction. The investigator proposes to extend existing CDM psychometric approaches to develop the diagnostic facet status model (DFSM) for formative assessment design based on the high school physics curriculum. There are a number of working concerns with existing CDMs. They do not scale well to high-dimensional facet spaces, they require complex coding schemes based on expert input, they cannot explore intricate facet relationships, or option-based nominal responses. This project will build on existing CDM approaches to address these limitations. The project develops a new psychometric model, the DFSM, and a longitudinal extension, which can simultaneously model high-dimensional goal and intermediate understandings at item option level. The PIs develop a machine learning method to identify relations among facets, yielding a comprehensive "facet map" that reveals both attribute hierarchies and conjoined facets for the learning of physics. Finally, the researchers conduct qualitative studies to validate DFSM and longitudinal DFSM's output for learning and instruction. The project is supported by NSF's EDU Core Research Building Capacity in STEM Education Research (ECR: BCSER) program, which is designed to build investigators' capacity to carry out high-quality STEM education research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.