SCH: Multimodal Techniques to Enhance Intra- and Post-operative Learning and Coordination between Attending and Resident Surgeons

Surgical errors have the highest risk of severe patient injury and death. High quality training of surgeons is an important element in reducing these harms, yet there is accumulating data showing that U.S. general surgeons are not adequately trained by the time they graduate from residency. The overarching goal of the project is to improve the training experiences and outcomes in the operating room by computationally modeling attending and resident surgeons' intraoperative behaviors. In particular, by modeling surgeons' individual and interpersonal behaviors, the project aims to enable automated inference and formative assessment of resident surgeons' technical competence, attending surgeons' instruction quality, and the degree to which residents gain operational independence. The project will also develop interactive systems at Michigan Medicine that will enhance resident surgeons' readiness for independent practice and attending surgeons' awareness and quality of teaching. The project will focus on four main activities. Thrust 1 will involve collecting and curating a dataset of up to 100 laparoscopic cholecystectomy surgeries. For each procedure, three streams of multimodal data will be collected: surgeons’ gaze, conversations in the operating room, and video feed of the procedure obtained through the laparoscopic camera. Thrust 2 will develop and evaluate a multimodal modeling approach to predict surgeons’ intraoperative behaviors. The approach will first encompass a scene segmentation pipeline to interpret surgery scenes and add semantic meaning to surgeons’ gaze, such as “the resident is looking at the gallbladder.” It will then build a multimodal neural network architecture to process different types of input data to predict two levels of intraoperative behaviors: 1) technical procedures and 2) communication and coordination behaviors. Thrust 3 will develop and evaluate a post-operative debriefing dashboard for surgeons to review critical moments in the surgery. The research team will co-design the dashboard with attending and resident surgeons and explore interaction techniques that enable surgeons to easily review moments of interest. Controlled experiments will be conducted to evaluate the effectiveness of the techniques. Thrust 4 will develop and evaluate augmented reality-based intraoperative visualizations to enhance coordination and instruction. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.