Multimodal, Multi-scale Framework for Ethical AI Model Development

Addressing healthcare disparities and improving precision medicine requires moving beyond race-based medical models. Current clinical algorithms often overlook the full spectrum of patient diversity, including biological, social, and environmental factors, leading to biased and inaccurate predictions. This is evident in diseases like breast and prostate cancer, where disparities persist despite advancements in treatment. Our proposal aims to address these gaps by developing novel digital phenotypes through the integration of multimodal datasets—imaging, pathology, electronic health records (EHR), and social determinants of health (SDOH). Leveraging advanced techniques in radiopathomics extraction, data fusion, novel graph architectures, and unsupervised clustering, we aim to discover biologically relevant patient populations to improve risk stratification and treatment optimization. To achieve this, we will first develop an open-source fusion framework for multimodal data integration. This robust framework will be incorporate tools for data harmonization, radiopathomics feature extraction, multimodal data embedding, and graph neural networks. We will apply this framework to discover biologically relevant digital phenotypes for breast and prostate cancer, and train phenotype- specific risk prediction models which will be robustly validated at 4 external sites. Our proposal emphasizes an ethical AI framework that incorporates privacy, fairness, accountability, and inclusivity principles throughout the model development process. We will adopt an iterative model co- development approach, engaging multidisciplinary stakeholders, including physicians, collaborators, and industry partners, to continuously refine our data and models. This co-design process ensures that our models remain clinically relevant, ethically sound, and aligned with the needs of diverse patient populations.