Collaborative Research: SHF: Medium: Fine-Grained Software Testing

Testing is the primary means of validating software correctness in practice, and developers often write tests at different levels of granularity. Specifically, unit tests validate individual functions, integration tests validate interactions among functions, and system tests validate end-to-end system behavior. This decades-old categorization of tests is valuable, evidenced by widespread tool and framework support, but it hinders developers from testing at finer granularity levels, such as statements within functions. Yet, many software bugs occur at such finer granularity levels and those software bugs often escape the forms of tests that are used today. This project aims to enable developers to perform fine-grained testing, thereby increasing software quality. Developers will be able to test hard-to-reach and hard-to-comprehend code fragments, complementing existing testing methodologies. The resulting higher-quality software is expected to contribute positively to the US economy. The research will be integrated into curriculum and training. The project's underlying research objective is to increase the efficiency and efficacy of software testing by removing decades-old artificial boundaries that exist between tests and code. To achieve this objective, this project will (1) develop a language and a framework for expressing and using fine-grained tests; (2) automatically generate fine-grained tests from code or existing tests, making it easier to retrofit them to existing code; (3) adapt fine-grained tests to software evolution and use fine-grained tests to improve current regression testing techniques; (4) use fine-grained tests to improve fuzzing and runtime verification; and (5) begin supporting fine-grained testing of non-functional properties, focused on specific security and performance bugs. Proposed techniques will be evaluated via experiments on open-source projects, to evaluate their ability to increase the coverage and bug-finding capability of existing test suites. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.