Multimodal Learning for Contextually-Aware Longitudinal PET/CT image analysis

PROJECT SUMMARY 18F-Fluorodeoxyglucose (FDG) PET/CT imaging has become an essential tool for guiding and adapting treatments for lymphoma. However, the PET evaluation criteria currently used for assessing lymphoma, which consists of subjective visual scoring on a 5-point scale, is suboptimal. The visual scores suffer from high inter- observer variability and have low prognostic power for new emerging biological therapies. Quantitative PET metrics have been shown to be more predictive of clinical outcomes than visual scores, but quantitative analysis of whole-body PET/CT images is prohibitively time-consuming and impractical in routine clinical care. Deep learning (DL) has shown promise in automating the quantitative analysis of baseline FDG PET/CT images, but comprehensive evaluation of interim-therapy and post-therapy images using DL has proven difficult. Residual lymphoma has low-level uptake, which can be hard to differentiate from physiologic or treatment-related uptake, and reading physicians must use clinical histories and baseline PET images (i.e., sites of initial disease) to make reliable diagnoses. DL algorithms, on the other hand, only operate on cross-sectional images and are unable to account for historical context. Our objective is to develop DL algorithms that operate on PET/CT images from more than one time point so that algorithms can learn longitudinal dependencies for contextually-aware predictions. We also aim to develop multimodal vision-language models that can simultaneously interpret radiology text reports while performing PET/CT image analysis. These models can leverage critical information about patient history and physician interpretation when processing retrospective images. Furthermore, we will use semi-supervised learning to leverage both unlabeled datasets and labeled datasets. Our overall goal is to develop contextually-aware algorithms for automated longitudinal analysis of whole-body PET/CT images in lymphoma. These tools will be developed using diverse datasets from multiple institutions. PET metrics measured by DL will be validated as predictive markers of outcome using data from a Phase 3 clinical trial.