Co-Design and Collaborative Development of an Imaging-Text-EHR Multimodal Foundation Model for Outcome Prediction of Critically Ill Patients

ABSTRACT Mortality and morbidity are the most essential outcomes in patients admitted to the intensive care unit (ICU). Timely prediction can aid clinicians in making treatment decisions, optimizing resource allocation, improving outcomes, and managing patients’ and families’ expectations. Previously, our research team developed a multimodal deep learning model predicting mortality and morbidity among ICU patients by incorporating structured data (time-variant and time-invariant electronic health record [EHR] data) and unstructured data (clinical text and chest X-ray images). In this project, we aim to develop a multimodal foundation model that considers these three data modalities: chest X-ray (imaging), clinical notes (text), and structured data (EHR). The foundation model will then be fine-tuned to predict the probabilities of outcomes (mortality, morbidities, etc.) after different periods of time (within 4, 12, 24, or 48 hours, for example). Explanation of the predictions will be generated leveraging the multimodality of the model to help users assess its trustworthiness. To ensure that the design will meet the needs of care providers and minimize any ethical concerns, a co-design approach will be applied where clinicians, patient family members, ethics experts, and computer scientists will collaborate and iteratively develop the model using feedback from stakeholder engagement. The approach is divided into three tasks: (1) stakeholder engagement to aid in ethical and model design; (2) development of multimodal models, including using a foundation model architecture; and (3) external validation of models and assessment of algorithmic bias. The model may serve as a generalizable architecture for other multimodal models predicting clinical outcomes.