SCH: Technology for Personalized Surgical Feedback using Vision and Language Data.

Surgeons need expert feedback to improve their skill throughout their career. But expert feedback is not easily available to surgeons after they complete training. Expert feedback typically takes the form of natural language, i.e., surgeons learn from experts’ verbal teachings. Recent advances in artificial intelligence (AI) have created a tremendous potential for technology that can provide surgeons with expert-like language- based (i.e., narrative) feedback on any surgery they perform. This project aims to leverage current AI models and develop new ones that analyze videos of surgical procedures and generate expert-like narrative feedback. The AI models will include those that analyze language (i.e., large language models) and videos plus language (vision language models). This project will focus on cataract surgery as a prototype procedure to develop the AI models and evaluate them in additional procedures including surgery of the sinuses around the nose and surgery to remove a lobe in the lung. The overall goal for this project is to develop AI models that provide an expert analysis of a surgical video that includes description, interpretation, and reasoning about what is observed in the surgery and prediction of how the procedure evolves over time. To achieve this goal, this project consists of the following specific aims: (1) To develop a unified framework of vision language and large language AI models to generate expert analyses of surgical videos; (2) To develop methods for the AI models to continuously learn from techniques such as data augmentation, pretraining, and incorporating expert feedback; (3) To develop methods for synthesizing surgical videos from expert analyses to address the challenges in creating sufficiently large datasets needed to train the AI models; and (4) To create a dataset that enables this research. The expected impact of this work is to allow surgeons to learn more skill quickly and reduce variation in patient outcomes resulting from different skill among surgeons. RELEVANCE (See instructions): Surgery is indispensable for public health and therefore, it is critical to enable surgeons to improve skill throughout their career. This project will develop artificial intelligence models that provide detailed narrative personalized feedback using vision and language like that provided by an expert surgeon. These models enable all surgeons to improve their skill through experience, which improves patient outcomes.