Multimodal machine learning for diagnosis and mechanistic phenotyping of inherited diseases

ABSTRACT Inherited diseases frequently manifest with distinctive facial features, collectively referred to as the “facial gestalt.” However, the effectiveness of using facial information relies heavily upon the clinician's breadth of experience, and such reliance often leads to delays in diagnosing patients. Furthermore, 1 in 600 newborns have craniofacial anomalies, impacting the skull, jaws, ears and/or teeth. Improved therapeutic approaches require a deep understanding of the molecular and biological systems underlying these craniofacial conditions. Through an interdisciplinary and agile team, our study has two objectives: (1) We will develop an ethically focused and data-driven multimodal machine learning algorithm that integrates frontal facial photos, clinical notes, patient demographics, and structured phenotypes to improve diagnosis of genetic syndromes. (2) We will develop a multimodal algorithm that takes 4D videos (non-invasive laser scanning to map craniofacial surface morphology), facial photos, electronic health records, and omics information (genome, transcriptome and epigenome), to prioritize genes in key pathways that are relevant to phenotypic features of craniofacial abnormalities. We will adapt a patient/clinician engaged design, ensuring that the training data and outcome evaluation metrics align closely with real-world applications. We will develop a modularized open-source software to decouple the multimodal approach from the underlying pretrained models, ensuring that the final product is adaptable to rapid changes in AI technologies to be easily integrated into evolving healthcare landscapes. Our project addresses the social-technological challenges in clinical diagnosis of rare diseases, to advance understanding, interpretation and prediction of complex biological, behavioral, and health manifestations through the implementation of multimodal approaches.