Planning: CIRC: Planning-C: A Holistic Benchmark for Accelerating Research in Large Language Models for Software Engineering

In recent years, Large Language Models for code (LLMc) have significantly transformed the landscape of software engineering and development. Tools based on LLMc, such as GitHub Copilot, have been widely adopted to assist software engineers in writing, analyzing, and debugging code. A crucial research question is how to assess the quality of the generated code. Currently, the evaluation process relies heavily on accuracy and robustness metrics, while neglecting other important factors such as interpretability, efficiency, bias, and fairness. This incomplete assessment may lead to significant social harm and bias. Therefore, this project aims to standardize testing scenarios for different LLMc and provide a multi-metric measurement system beyond a single accuracy metric. The main contribution of this project is to create an infrastructure for the benchmarking and holistic evaluation of Large Language Models for code. The investigators will carry out the following research and outreach activities: (1) conduct a survey with the software engineering and machine learning communities to gather requirements and gauge interest in such an infrastructure; (2) conduct interviews to understand the barriers researchers face and how a community infrastructure and benchmarks might mitigate these barriers; (3) host a workshop to facilitate the exchange of best practices used by researchers for evaluating LLMc and to understand how a community infrastructure and dataset can assist researchers; (4) conduct a working group to explore the design of the infrastructure with researchers interested in LLMc; (5) conduct a summative survey to assess the capability of our proposed infrastructure to accelerate LLMc research. Overall, this project will benefit both the software engineering and machine learning communities and fundamentally advance research on LLMc. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.