NSF Safe-OSE: Scalably Detecting Inconsistencies Between Git Commit Messages and Code in Open-source Projects

Computer systems research relies on software simulation to perform early-stage design-space exploration, quickly iterate on new hardware-software changes, and explore the viability of new ideas before committing to expensive hardware prototyping and fabrication. In computer architecture, gem5 is the most widely used open-source simulation tool, heavily employed by academia, industry, and national laboratories. Like many other open-source projects, gem5 relies on a small number of volunteer maintainers to manage the ever-growing numbers of users and proposed changes. These maintainers typically have no formal training in identifying potential security vulnerabilities. Moreover, computer architecture designs prototyped in gem5 are often incorporated directly into real hardware. Vulnerabilities in gem5 could thus cause practitioners to inadvertently introduce security vulnerabilities into hardware that cannot easily be patched, impacting future commercial hardware and threatening national infrastructure. This project provides a novel framework to strengthen the security of the gem5 codebase and any future hardware designed with it. This project will also increase the impact of results generated with gem5 for U.S. industry and national lab practitioners by allowing them to more safely use the cutting-edge, top-secret designs they prototype in gem5. This project combines expertise across simulator design, computer architecture, programming languages, security, and language modeling to develop and deploy VULCAN (VULnerability Consistency ANalyzer). VULCAN is a framework that detects and mitigates security vulnerabilities before they are introduced to the gem5 repository. Specifically, VULCAN detects inconsistencies between commit messages and code changes, and discovers known hardware vulnerabilities. Overall, VULCAN comprises four components: 1) leverage of Large Language Models (LLMs) and static analysis to synthesize formal specifications from code diffs; 2) utilization of Natural Language Inference (NLI) to check consistency between the commit message and the synthesized code specification; 3) integration of a taxonomy of hardware vulnerabilities, which it uses to automatically detect if code changes introduce known issues; and 4) the design and integration of a user interface into gem5’s GitHub code review process, enabling maintainers to seamlessly consume findings from the other components. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.