Collaborative Research: SaTC: CORE: Medium: Securing LLMs against Prompt Injection Attacks

Large Language Models (LLMs) are increasingly deployed as the backbone of real-world applications such as Google Search with AI Overviews and Microsoft Bing Copilot. When data and code are not properly separated within an application, the latter (including AI applications) is vulnerable to cyber-attacks. This project's novelties are twofold: (1) conducting a systematic study to deepen the understanding of such threats, and (2) developing new defenses to mitigate such attacks. Its broader significance and importance lie in establishing foundational security principles for the rapidly growing ecosystem of AI applications, which are now widely deployed across diverse societal domains. Moreover, the released code and materials produced by this project will not only help secure real-world LLM-integrated applications but also serve as valuable educational resources for undergraduate and graduate courses, fostering the next generation of researchers and practitioners in this emerging security area. Security history shows that when data and instructions are not properly separated within a system, injection attacks can emerge—for example, SQL injection attacks in traditional software. Similarly, due to the lack of a clear boundary between instructions and data in prompts, LLM-integrated applications are inherently vulnerable to prompt injection attacks. To understand and mitigate such threats this project adopts a holistic approach comprising three interconnected research thrusts to systematically investigate the security vulnerabilities of LLM-integrated applications to prompt injection attacks and to develop new methods to prevent, detect, and attribute such attacks. The project will also open-source a platform that integrates our developed algorithms along with a comprehensive tutorial on prompt injection attacks and defenses. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.