III: Building Reliable Retrieval-Augmented Reasoning Agents for Health Applications

Large language models (LLMs) fundamentally transform the way humans seek information from the internet. We have witnessed a paradigm shift from traditional search engines, which return lists of documents for users to synthesize to generative engines that can synthesize existing knowledge to compose effective answers. This transition is largely powered by retrieval-augmented generation (RAG), which grounds model outputs in external documents to ensure factuality. Despite recent advances, building effective autonomous search agents for health applications remains a significant challenge. This project develops a framework for dynamically retrieving external knowledge during the search and explicitly integrating that knowledge into the reasoning process. The approach enables systems to combine the knowledge encoded in LLMs with up-to-date and domain-specific information held in external sources. This project addresses limitations of the current approaches, reduces LLM hallucination, and supports multi-step, explainable, and robust inference for health applications. This project has a significant impact in advancing the national health by designing customized health applications, such as evidence-grounded clinical decision support, personalized medicine, customized remote support for chronic disease self-management, and clinical trial matching and eligibility explanation. To build an agentic-RAG framework for optimizing RAG-based reasoning, this project conducts the following innovative research: (1) the project introduces a new agent architecture called Re+Search (referring to Reason, Reflection, Search) for reasoning reflection as a powerful but underexplored capability for search agents, (2) the project formulates the knowledge-intensive health question-answering as a high-level Markov decision process (MDP) with well-defined intermediate actions, and provides a modular approach for optimizing search agents across three key dimensions, including the Re+Search agent with reasoning reflection, parameter optimization through fine-grained step-level supervision, and action evaluation using trained critics, and (3) the project develops a framework to enhance the faithfulness of RAG outputs, which can detect unfaithful RAG outputs using LLM internal activations and mitigate hallucinations for reliable agent reasoning. This project improves agentic RAG systems by emphasizing process-level supervision, rather than focusing solely on outcomes, providing a robust foundation for developing reliable information-seeking agents. By framing retrieval-augmented generation as a Markov decision process in which intermediate actions are both observable and evaluable, the agentic-RAG framework enables direct optimization of the search process. The modular design of the agentic-RAG framework offers a practical approach to deploying these systems in complex health-related applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.