SaTC 2.0: RES: Securing Retrieval-Augmented Generation of LLMs

Retrieval-augmented generation (RAG) augments large language models (LLMs) by grounding their outputs in documents retrieved from external databases. However, RAG systems can be compromised when their external databases are manipulated by attackers. This project’s novelties are twofold: (1) performing a systematic study to understand the security vulnerabilities of RAG systems, and (2) developing defenses to prevent, detect, and mitigate such security vulnerabilities. The project's broader significance and importance are to build the security foundations for RAG systems to enable their secure and beneficial integration into our society across diverse domains, especially in reliability-critical fields such as medicine, finance, law, science, and education. The reliance of RAG systems on external databases introduces a new attack surface: an attacker can manipulate the output of a RAG system by injecting malicious documents into the databases. The ultimate goal of this project is to secure RAG systems from such attacks. To this end, the project develops a holistic approach across three interconnected research thrusts. The first trust systematically uncovers security vulnerabilities of RAG systems. The second and third thrusts design defenses to prevent and detect attacks, as well as to recover RAG systems from attacks. Research outcomes are disseminated by wrapping up the developed methods into a Python package for integration into popular RAG frameworks. The project also produces educational resources for undergraduate and graduate courses and K–12 outreach activities, fostering the next generation of professionals. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.