Multimodal and Generative AI for Pathology

Abstract The microscopic examination of stained tissue is a fundamental component of biomedical research and for the understanding of biological processes of disease which leads to improved diagnosis, prognosis, and therapeutic response prediction. Ranging from cancer diagnosis to heart rejection and forensics the subjective interpretation of histopathology sections forms the basis of clinical decision making and research outcomes. However, it has been shown that such subjective interpretation of pathology slides suffers from large interobserver and intraobserver variability. Recent advances in computer vision and deep learning has enabled the objective and automated analysis of images. These methods have been applied with success to histology images which have demonstrated potential for development of objective image interpretation paradigms. However, significant algorithmic challenges remain to be addressed before such objective analysis of histology images can be used by clinicians and researchers. In this R35 renewal leveraging extensive experience in developing and disseminating research software based on deep learning the PI and his team will pioneer novel algorithmic approaches to address these challenges including but not limited to: (1) unimodal and multimodal self-supervised model for computational pathology (2) fundamental redesign of data fusion paradigms for integrating information from microscopy images and molecular profiles (from multi-omics data) for improved diagnostic and prognostic determinations (3) developing visualization and interpretation software for researchers and clinical workflows to improve clinical and research validation and reproducibility. The system will be designed in a modular, user-friendly manner and will be open-source, available through GitHub as universal plug-and-play modules ready to be adapted to various clinical and research applications. We will also develop a web resource with pretrained models for various organs, disease states and subtypes these will be accompanied with detailed manuals so researchers can apply deep learning to their specific research problems. Overall, the laboratory’s research will yield high impact discoveries from pathology image analysis, and its software will enable many other NIH funded laboratories to do the same, across various biomedical disciplines.