Collaborative Research:SCH:Bimodal Interpretable Multi-Instance Medical-Image Classification

This project focuses on creating a smarter artificial-intelligence (AI) system to better understand and analyze complex medical images, such as those from multiple scans of a patient. Traditional methods have had some success but face challenges in dealing with rare diseases and in providing explanations that doctors and patients can easily understand. This project aims to develop a modern AI approach that overcomes these limitations by leveraging a vast collection of medical images and doctors' notes, regardless of the specific health conditions to which they pertain. The research team will tackle various challenges to make the AI system more scalable, interpretable, and robust. This innovative project will deliver trustworthy AI-driven diagnostic tools to medical workers, expediting the diagnostic process for complex medical images. The impact of this project will be felt broadly in AI research and beyond, as its foundational research is likely to have impact in various applications, and its use-inspired research will enable the accelerated transition of modern AI approaches into benefits for society. The approach to achieve the overarching goal is to develop a bimodal interpretable multi-instance medical image classification framework by a scalable pretraining and finetuning approach. The framework consists of bimodal prototype-based interpretable contrastive pretraining to learn paired image and text prototypes from imbalanced unlabeled data, and multi-instance learning by deep area-under-the-receiver-operator-curve (AUC) maximization methods to learn from imbalanced patient-level labeled data. To make contrastive pretraining scalable and robust to imbalanced data, the investigators will develop a unified framework based on partial AUC losses, which not only unifies the existing contrastive loss but also induces new advanced global contrastive losses. The team of researchers will leverage new optimization tools and develop improved stochastic algorithms with mathematical guarantee without dependence on the large batch size of existing methods. To make multi-instance learning scalable and robust to imbalanced data, the investigators propose efficient stochastic algorithms for multi-instance deep AUC maximization by developing stochastic pooling operations from the lens of multi-level compositional optimization. The investigators will not only employ standard performance metrics for evaluation but will also leverage the domain expertise from radiologists to evaluate model performance and interpretability. The investigators will disseminate results through publications, open-source software, tutorials, workshops, and course materials, additionally engaging in outreach initiatives to enhance STEM learning and foster greater interest in the field. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.