Natural language processing and medical imaging analysis for multi-modality computer assisted diagnosis of ophthalmic diseases

The goal of this study is to develop and evaluate multi-modality artificial intelligence (AI)-assisted diagnosis models. Timely and accurate diagnosis of diseases for early intervention is key to maximizing patient outcomes. This is particularly true in ophthalmology, where timely treatment is crucial for minimizing vision loss. In recent years, AI has been widely applied in medical specialties such as ophthalmology through computer-assisted diagnosis, which provides scalable disease diagnosis with the potential for screening patients earlier. Yet, most methods use a single imaging modality for diagnosis, which fails to capture other modalities such as different imaging types, patient symptoms, medication history, or lab results information documented in electronic health records (EHRs) that are critical for disease diagnosis. Another important concern is the generalization capability of AI models. For example, only a limited number of studies have conducted external validations, and the impact of factors such as age, race, and gender on the generalization of AI models remains largely unknown in ophthalmology. The specific aims are to (1) propose NLP methods for clinical concept recognition and normalization from clinical notes; (2) develop multi-modality AI models that use medical images, free-text notes, and structured information for ophthalmic disease diagnosis; and (3) benchmark and investigate potential solutions to improve the downstream accountability of AI models, including external validations, subgroup analysis, and generalization to other diseases beyond ophthalmology.