EAGER: Knowledge-guided neurosymbolic AI with guardrails for safe virtual health assistants

This project addresses the limitations of generative artificial intelligence (AI) systems, particularly in the context of virtual assistants used to support healthcare (VHAs). While VHAs show potential for empowering patients and addressing clinical expertise shortages, concerns about safety and accessibility arise due to inaccuracies in their outputs and their lack of adherence to the relevant standards of care. To mitigate these concerns, the project proposes an innovative approach for integrating clinical protocols and practice guidelines within AI systems. This approach will enable the development of safety constrained VHAs that support clinicians and ensure safe interactions with patients. Additionally, the approach facilitates the provision of clinician-friendly explanations, fostering improved collaboration between humans and AI in healthcare. By addressing significant current concerns surrounding the safety of generative AI, the research will promote user confidence and adoption in safety-critical domains requiring human-AI collaboration. The research's success can have implications beyond healthcare, such as autonomous vehicles incorporating traffic rules or manufacturing processes ensuring safe operations and maintenance compliance. Furthermore, the project aligns with efforts to promote inclusivity in computing, workforce development, and education. Example initiatives include annual AI summer camp for school students from underrepresented backgrounds, and engagement with high school, undergraduate and graduate students through internships and workforce development modules relevant to interdisciplinary AI careers. The main innovation of this research lies in leveraging Knowledge Graphs (KGs) to construct guardrails that help ensure the safety of AI systems. In collaboration with clinical experts, a KG enriched with both declarative (e.g., medical terminology and definitions) and procedural or process knowledge (e.g., diagnostic criteria and clinical practice guidelines) will be employed to guide neural processing architectures, resulting in the development of VHAs inherently constrained to be safe. Furthermore, the same proposed methods will also equip VHAs with the ability to generate end user (e.g., clinician) friendly explanations making the system verifiable. The project's two important outcomes will be to (a) effectively apply medical guidelines from the KG to uphold high safety standards in a clinical setting, and (b) generate explanations that are easily comprehensible to end users using the terms, concepts, and guidelines relevant to end-user verification and decision-making. The core techniques proposed in this project will advance the state-of-the-art in neurosymbolic AI toward facilitating robust (verifiable and safety-constrained) collaboration between humans and AI. These advances have the potential for transferability to other domains with safety-critical applications, thus contributing to the broader field of AI research and its wider adoption. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.