CAREER: Attending & Bridging Context In Vision-Language Modeling For Medical Image Analysis

Artificial intelligence is increasingly used to help doctors interpret medical images and write clinical reports, yet today's most capable systems share three weaknesses that limit their value in real healthcare settings. They are computationally expensive, requiring infrastructure that is unavailable in many clinics. They fail when patient information is incomplete, which is a common occurrence in everyday medical practice. And their internal reasoning is difficult for clinicians to inspect, making it hard to trust their outputs in high-stakes decisions. This project will develop a new generation of medical artificial intelligence that is robust, efficient, and interpretable. The work will produce models that learn to separate what they see in a medical image from what they read in a clinical note, while also recognizing what is common to both. This separation allows the system to continue producing accurate, coherent reports even when clinical text is missing or limited, and to explain which image regions influenced its conclusions. The methods will be tested on three high-impact medical applications: automated reporting for eye disease, automated reporting for chest radiographs, and prediction of cognitive and behavioral development in adolescents using one of the largest pediatric brain imaging studies ever conducted. The project also creates educational opportunities for high school students, undergraduate and graduate students, and practicing clinicians, with a focus on broadening participation from groups underrepresented in computing and biomedical research. By advancing trustworthy artificial intelligence for healthcare, this project supports national priorities in artificial intelligence, biotechnology, and translational science, and contributes to improving public health through more accurate and accessible diagnostic tools. The technical objective of this project is to develop disentangled and adaptive vision-language models for medical data analysis. Three coordinated research challenges will be addressed. First, the team of researchers will design dynamic and adaptive fusion mechanisms that move beyond static cross-attention. A new Abstractor module will enhance vision and language features with cross-modal context before fusion, and a new Adaptor module will use a learnable modality indicator to weigh visual and linguistic information on a per-case basis. Second, the project will formulate principled disentangled representations through a Vision-Language Variational Autoencoder that factorizes the latent space into vision-specific, language-specific, and shared components, regularized by orthogonality and contrastive alignment constraints. Recent peer-reviewed work by the investigator and his team, presented as an oral paper at a top artificial intelligence conference in early 2026, has already established the feasibility of this factorization and demonstrated robust radiology report generation when clinical indication text is withheld at inference. Third, the project will validate the integrated framework across three diverse domains: retinal image captioning on a public ophthalmology benchmark, chest radiograph report generation on standard radiology benchmarks with systematic missing-modality evaluation, and prediction of developmental trajectories from structural and functional magnetic resonance imaging (MRI) in the Adolescent Brain Cognitive Development study. A compact, parameter-efficient language decoder will translate the learned representations into clinical text without the computational overhead of much larger vision-language models. Educational activities include a new graduate course on artificial intelligence in medicine, summer workshops for high school students, short courses for clinicians, and an undergraduate and graduate research training program that leverages the unique student diversity of the host institution. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.