Ethical Multimodal AI Development for Obesity, Diabetes and Digestive Disease

Abstract Multimodal AI (MAI) has enormous potential to provide useful tools for AI-integrated healthcare. To realize this potential, there is a need to address difficult problems at multiple stages of MAI model development: 1) identify and prepare clinically-relevant, inclusive datasets to assure generalizability, 2) develop ethical MAI models integrating multimodal data, 3) develop MAI models that account for missing data, longitudinal data, and distribution shifts, 4) co-design models with stakeholders and 5) develop green, scalable compute infrastructure and models that can be deployed in practice. We will address these challenges in the following aims. In Aim 1, we will develop self-supervised MAI foundation models for multimodal and longitudinal input/output data. Data, such as images, videos, electronic health record data (structured/unstructured text), biospecimen and genetic data will be encoded into a shared representation space. Our novel model will include a virtual ethics critic to supervise model expressiveness along ethical guidelines. The data source is the Penn Medicine BioBank (PMBB), which has enrolled 250k+ patients. In Aim 2, we will use an iterative and concurrent mixed methods co-design evaluation, including a multi-stakeholder committee and interviews with providers and patients informed by a conceptual framework for the ethical design, use and governance of AI in healthcare to inform model construction in Aim 1. In Aim 3, the general-purpose model will be fine-tuned for liver disease, digestive disease and frailty applications. The deliverable will be open-source MAI models, employing FAIR principles and informed by ethical co-design to provide clinical decision support.