SaTC: CORE: Small: Measuring and Improving Reproducibility

Advances in science do not come solely from novel exploratory studies. Reproducibility, or the ability of a third party to substantially recreate the same results of a previous experiment using the same code, conditions, and input data, is therefore crucial as a means of closing the gap between initial discovery and widespread adoption of claims. However, the discipline of computer security has historically performed poorly in this regard. Improving reproducibility in computer security is not only important in making the results of research more trustworthy but also has the potential to speed up efforts by other scientists to build on results, and to take their results out of the laboratory and into practice via commercialization or open-source. This project has two main activities. First, the project team is performing large-scale reproducibility studies across important venues in computer security, characterizing how previously published efforts achieve or fail to allow for reproducible science. Next, the team is focusing on why such efforts fail and is creating improved frameworks for expressing gaps and limitations in the use of artifacts. The team is creating a framework to more readily characterize replicability, wherein artifacts are more broadly profiled using new data sets. These efforts will not only provide the community with benchmarks and guidance for improved performance but will also assist in making the results of the community's research efforts more readily transitioned to the world beyond research. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.