EAGER: Developing Concept-based Reasoning Approaches for Health

The success of deep neural networks (DNNs) and large foundation models such as vision language models (VLMs) and large language models (LLMs) has called for incorporating them in sensitive domains, such as medical applications. However, widespread clinical adoption remains extremely limited. Prototypes of these systems suffer from fundamental problems, such as the black-box nature of AI models; that is, a lack of transparency and interpretability. One critical issue that holds back the widespread adoption of machine learning in healthcare is the lack of accountability in the system's predictions. To make these models useful for healthcare, these systems must be designed with a bedrock of solid accountability and sound rational decisions before focusing on raw performance metrics. This EArly-concept Grant for Exploratory Research (EAGER) project develops concept-based reasoning approaches to improving interpretable DNN model design and improving rationales in large pre-trained models. The concept-based reasoning attempts to learn high-level ‘concepts’, which are abstract entities that align with human understanding and provide explicit rationales for model predictions to achieve interpretability. This EAGER project conducts two innovative research tasks: (1) Build inherently explainable concept-based DNN Models for Health. Concept learning models attempt to learn high-level concepts, abstract entities that align with human understanding, and thus provide interpretability to DNN architectures. This approach can be effectively utilized in domains such as medical diagnosis where concepts are sometimes undefined, for example, shape of bone in an X-ray where the shape itself is undefined, but a doctor can easily tell it is irregular. (2) Improve the pre-trained foundation models’ trustworthiness by incorporating concepts as structured entities that provide human-understandable reasoning. This task designs a neurosymbolic framework which provides concept-based reasoning through learning visually grounded concepts, linked neurosymbolically through rules. The framework is particularly effective on medical data, which is typically underrepresented in the pre-training of the large VLMs. A hierarchical concept tree structure is designed to provide a symbolic reasoning process through a vast search space created using knowledge banks to improve VLM predictions during inference. This research has the potential to have a significant impact in designing state-of-the-art medical diagnostic systems. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.