CAREER: Regularizing Large Language Models for Safe and Reliable Program Generation

With computing woven into almost every aspect of business and society, there is an increasing demand for various programming skills to drive economic growth and keep the nation competitive in technology innovation. However, programming is cognitively demanding and hard to learn. Recent advances in Large Language Models (LLMs) such as ChatGPT have demonstrated a great potential to address this challenge by generating complex programs from natural language, eliminating the need to memorize and grapple with program syntax and semantics. Despite this breakthrough, as a statistical method, LLMs suffer from a number of issues when applied to domains like program generation that emphasize functional correctness, safety, and robustness. For instance, recent studies have shown that LLMs may generate code that does not align with user intent, has security vulnerabilities, or violates coding standards. These issues consequently diminish programmer productivity and trust in LLMs. This project will advance the understanding of the limitations of LLMs in program generation and develop principled regulation approaches to enhance the correctness, safety, and robustness of LLM-generated code. The research findings and education activities of this project will inform the design of new instructional materials and pedagogical approaches to prepare computer science students for future careers in the age of large language models. This project follows a mixed-methods research design that combines empirical data-driven studies, algorithm development, and tool building. While there have been many recent efforts in evaluating and improving LLM-based program generation, it remains unclear what types of program generation errors LLMs make, whether different kinds of LLMs make different types of errors, and why they make such errors. This project will bridge the knowledge gap by conducting an in-depth analysis of the symptoms and characteristics of program generation errors made by various LLMs using grounded theory and developing root cause analysis methods to investigate the correlation between program errors and the internal states of LLMs. In addition, the project will develop new error localization and mitigation methods that leverage the internal states of LLMs to precisely target the root causes of different kinds of errors at a low cost. The project will also make the first attempt to mitigate non-functional concerns in LLM-generated code by developing new lightweight model adaptation methods. With these advancements, this project aims to fundamentally chang the way LLMs are trained and updated for safe and reliable code generation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.