SHF: Small: E2R2--A Comprehensive Approach to Improve Simulation-based Testing of Autonomous Driving Systems

Autonomous Driving Systems (ADS) are software systems designed to reduce or replace human involvement in driving vehicles. Improving ADS safety is critical to achieving road safety. Physical testing of ADS-driven vehicles, albeit important, is limited. Testing autonomous vehicles on city roads does not scale and, more importantly, can be unsafe. For these reasons, simulation-based testing has been shown to be of fundamental importance to ADS quality assurance; it enables developers to assess ADS robustness before deployment. Simulation-based testing relies on high-fidelity simulators. Developers “plug” their ADS into a simulation platform and run tests largely consisting of a route definition within a map with static and dynamic obstacles. Despite recent advances in techniques for simulation-based ADS testing, three important challenges remain. First, finding failure-revealing inputs using simulation is costly. Second, existing techniques often report duplicate infractions that do not contribute to information gain. Third, existing techniques make unrealistic assumptions about the environment. For example, they assume a single ADS-driven vehicle in the simulation. This project proposes novel approaches to mitigate these fundamental challenges, advancing the state of the art in simulation-based ADS Testing. In this project, machine learning-based techniques will be used to address the fundamental limitations of simulation-based testing for ADS. One research goal is to improve the efficiency and effectiveness of simulation-based testing by leveraging the risk signals produced by anomaly detectors during a simulation to improve the ability of simulation-based testing to detect infractions within a given time budget. A second research goal is to improve the relevance of testing by leveraging semantic information embedded in parts of a simulation to deduplicate infractions that testing techniques report. A third research goal is to improve the realism of simulation-based testing by using real-life data and human input. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.