CPS-FR: A Statistical Framework for Repeatable CPS Performance Assessment

Modern transportation, manufacturing, and service technologies increasingly rely on software, sensors, and artificial intelligence to make decisions and act in the physical world. The public, manufacturers, and regulators depend on performance tests to determine whether these systems are safe and capable. However, repeating the same test can produce different conclusions even when the system, task, and testing conditions appear unchanged. These inconsistencies may arise from sensor noise, environmental variation, randomized computation, and complex interactions between software and physical processes. Conducting many additional tests is often too costly, slow, or unsafe when real machines are involved. This project will establish scientific foundations for testing methods that produce consistent and statistically defensible conclusions while making efficient use of physical experiments. More dependable testing can improve safety decisions, strengthen the evidence used in standards and certification, reduce wasted testing resources, and support the responsible introduction of intelligent technologies. These advances serve the national interest by helping protect people, improving confidence in critical automation, and strengthening United States leadership in trustworthy engineering. The project will also provide open-source tools, shared data, and reporting practices for researchers, industry, and standards organizations. Its results will be incorporated into university courses and student projects to prepare future engineers to evaluate complex systems with greater rigor and transparency. This project studies repeatability as a property of the testing procedure rather than merely as a property of the system being tested. A testing procedure takes a system, an operating environment or task, and a set of test conditions and produces a performance result. The research will develop procedures for which independent executions under nominally identical conditions produce the same conclusion with a prescribed probability, while also maintaining accuracy and limiting the required number of trials. The first research component will address unitary systems, for which the testing procedure has access only to controllable inputs, stochastic disturbances, and observable outputs. It will extend preliminary methods for numerical measures, such as failure risk, to confidence intervals and other set-valued results, pass/fail decisions, and rankings of multiple systems. The methods will combine structured representations of possible outcomes with sequential statistical analysis, confidence bounds that remain valid as evidence accumulates, and adaptive rules for determining when sufficient testing has been completed. The second component will examine compositional systems built from interacting modules and determine how uncertainty in testing results propagates through sequential perception-planning-control pipelines, parallel sensor-fusion and multi-agent structures, and combinations of these architectures. The third component will translate the resulting theory into standardized reporting formats and open-source software for recording, monitoring, visualizing, and scheduling tests. The methods will be evaluated against conventional and fixed-sample testing approaches using physical humanoid and quadruped robots, industrial and open-source manipulators, and simulation environments for robotics and automated driving. The expected outcomes include mathematical guarantees, practical testing algorithms, interoperable software, benchmark data, and methods for incorporating statistical repeatability into engineering standards. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.