Collaborative Research: III: Knowledge Augmentation Towards Generalizable Model Editing

Foundation models, such as large language models (LLMs), are increasingly used to answer questions, interpret images, and support decisions in domains such as healthcare, science, and geospatial analysis. However, their stored knowledge can become outdated, incomplete, or incorrect as the world changes. Existing model editing methods can update specific facts, but often fail to generalize beyond the exact edit, apply the update in related contexts, or maintain consistency with dependent facts. This project investigates how to make foundation model editing more reliable and generalizable through knowledge augmentation. A successful outcome will advance the understanding of how foundation models store, update, and use knowledge, while supporting safer and more trustworthy applications. The education and outreach activities will train undergraduate and graduate students in reliable foundation models, multimodal learning, and responsible machine learning, with activities designed to broaden participation in computing. The primary goal of this project is to develop a knowledge augmentation framework for generalizable model editing in foundation models. The project studies knowledge augmentation during both model training and model inference. For training-time editing, the project will augment visual, linguistic, and graph-based knowledge edits using visual generation, text augmentation, and graph-guided identification of dependent facts. For inference-time editing, the project will develop methods that help models use edited facts more faithfully through contrastive decoding, preference-based optimization for context faithfulness, and reasoning models for multi-step question answering. The project will evaluate the proposed methods on text-to-image generation, question answering, classification, and completion tasks, with applications in medical image diagnosis and geographic remote sensing. The resulting methods and software are expected to help researchers and practitioners update foundation models more accurately, preserve unrelated knowledge, reduce hallucinations, and improve reliability in knowledge-intensive applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.