Planning: AI-Ready: TOPSAIL: The AI-Ready Test Bed in Education

In education, artificial intelligence (AI) enabled tools and applications are rapidly emerging, which show strong potential to improve student outcomes. However, the speed of innovation has outpaced our ability to evaluate these improvements. When implemented in a student learning environment, only some of the AI-enabled applications will be effective in improving student learning, performance, and success. Others may even harm learning. If the US education system is going to reap the benefits of emerging AI applications, the education research community needs novel infrastructure to evaluate these benefits and to inform education leaders and the educational technology marketplace about their effectiveness. Furthermore, AI-enabled learning technologies raise new concerns about privacy and safety that need to be addressed when conducting real-world evaluations with students. This project will lay the groundwork to create TOPSAIL (Testing Outcomes Privately and Securely for Artificial Intelligence in Learning), a testbed that enhances existing education research infrastructure and provides an expressway for safe and secure impact assessment of AI innovations in real classes. The TOPSAIL testbed will be situated in the Canvas learning management system, and will leverage Terracotta, an emerging open-source experimental research plugin, to support a range of ethical and privacy protections for students as research participants. Canvas, made by Instructure, is the most widely used learning management system in the US, and provides a well-established and versatile platform for testbed research. The current planning project will assemble a team including research scholars, industry leaders, and administrators at three school districts, who will co-create the design requirements for TOPSAIL. These requirements will address concerns about AI safety and privacy, in tandem with more general concerns about digital platform-based research with students. These concerns will be surfaced and documented during on-site focus groups with school stakeholders at the three districts. Focus group protocols will direct attention to known issues in education research (safety, privacy, data collection, experimental control, and collaborative involvement of teachers). Once these concerns are surfaced, solutions will be co-designed during a multi-day in-person workshop with the full planning team, and then synthesized into a detailed set of design requirements for TOPSAIL. The project will allow us to build a coalition of trust among involved stakeholders, resolve key concerns, and set a collaborative path forward for testbed development. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.