CAREER: Building Next-Generation Language Models Based on Retrieval

Large language models (LMs) have revolutionized the field of natural language processing, achieving state-of-the-art performance in a wide range of downstream tasks. Despite the success, these LMs are trained on enormous amounts of text data and cost a massive amount to create and run. Additionally, they are inherently difficult to interpret, challenging to update with ever-changing real-world information, and may leak private user information. This proposal seeks to develop an alternative paradigm to standard language modeling: retrieval-based language models, with the aim of reducing training and inference costs while also providing benefits such as better interpretability, adaptability, and privacy. This research will be integrated into education through new teaching modules in developing undergraduate and graduate natural language processing courses, promoting education for undergraduate research, and outreach to K-12 students and teachers from underrepresented communities. This project addresses a full pipeline including training, scaling, adapting, and using retrieval-based language models and is organized into four components, including (1) building a general learning framework for retrieval-based LMs and developing scalable algorithms to support end-to-end learning; (2) investigating the scaling law of retrieval-based LMs and developing better data quantization to improve inference efficiency; (3) devising methods to quickly update retrieval-based LMs and adapt them to unseen and privacy-sensitive domains; (4) designing effective approaches to use retrieval-based LMs on downstream tasks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.