CAREER: Automated Multimodal Learning for Healthcare

Multimodal learning is one of the central tasks of artificial intelligence (AI), which aims to effectively fuse and model multimodal data to gain a better understanding of the world around us. Many multimodal fusion strategies have been proposed, ranging from manually designed policies to advanced automated machine learning (AutoML)-based approaches. Although AutoML-based solutions outperform handcrafted ones, they are still far from optimal due to their lack of generalizability in model design and failure to account for the unique characteristics of multimodal data. This project takes the multimodal healthcare predictive modeling task as a representative example, aiming to discover and identify the optimal way to fuse multimodal data via a new learning paradigm, i.e., automated multimodal learning, with minimal human interventions. The success of this project will yield new fundamental knowledge in various fields, including automated machine learning, multimodal deep learning, and healthcare predictive modeling. The new automated multimodal learning paradigm will revolutionize multimodal data mining by automatically searching for new and complex yet optimal fusion strategies from the data, potentially motivating researchers and domain experts to understand the multimodal data better. In addition, recognizing unique research challenges posed by the unique nature of multimodal data in the healthcare domain and providing customized solutions will advance the research of healthcare predictive modeling significantly. To meet these goals, the investigator proposes to equip automated multimodal learning with the ability to model the unique challenges of multimodal health data, including data size variety, noise, and missing modalities. The investigator also proposes to validate the proposed research for different multimodal fusion tasks in healthcare informatics and beyond and gather feedback from experts to refine the proposed research. The results of this project will provide a needed paradigm shift toward automated multimodal data fusion, impacting a broad range of research fields, including machine learning, data mining, and healthcare informatics. The proposed research will also make an enduring contribution to multimodal predictive modeling in clinical practice and other domains. The generated data, source codes, and software tools will be made available to researchers worldwide. The open platform will expedite research, enhance global collaborations in this field, and provide longstanding value for academia, healthcare organizations, and health industries. The proposed education plan will help to ensure that graduates are well equipped to design and evaluate machine learning solutions and cultivate K-12 students' interest in computer science and informatics. It will also lead to a growing population of undergraduate research assistants and enhance collaboration and networking among graduate students. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.