Autonomous Gaze Coaching for Virtual Reality Telesurgical Simulators

Project Summary The proposed project aims to develop the first-ever data-driven autonomous gaze coach for robot-assisted minimally invasive surgical (RAMIS) virtual reality simulation training. A surgeon’s skill level is correlated with patient outcomes. It is known that expert surgeons adopt different gaze patterns from novices, and that gaze training is a proven, but labor-intensive coaching method that improves a trainee surgeon’s surgical skills, especially under stress. Our proposed coaching system will increase the accessibility of high-quality gaze coaching, even in areas where there is a shortage of expert instructors, and will free up valuable time for surgical experts for clinical and higher-order instructional work. The acceleration of skill acquisition, will, in turn, decrease the time and cost for training competent surgeons across the country. The project will comprise three specific aims. In Aim 1 we collect gaze and performance data from experts and novices in six virtual reality RAMIS drills. We will analyze the data to find good candidate drills in which to develop our autonomous gaze coaches by identifying those in which expert performance and gaze patterns differ most from those of novices. In Aim 2, we use the data collected in Aim 1 to train expert gaze-synthesizing neural networks that can generate a fixation point or a gaze map given surgical video streams. We validate the networks’ abilities to replicate expert gaze using a held-out test set. In Aim 3 we develop approaches to display the gaze synthesized by the neural networks from Aim 2 and test the effectiveness of our gaze coaching system to improve the surgical skills of novices. Outcome skill measures of this user performance study are the surgical skill measures such as economy of motion and the completion time, and task-specific penalties. These include instrument and apparatus collisions, dropping of apparatus, instruments out-of-view, durations of excessive force application, wrong energy application, dissection outside of defined zones, and repeated needle pierces. Outcome gaze measures include the increase in the amount of overlap between the novice gaze and expert gaze after training, and the increase in time spent focusing on expert-defined areas of interest in the surgical scene. The results of this development and validation of the system in virtual reality simulation will inform our long-term goal of developing autonomous gaze coaching systems for use in more complex clinical-like training scenarios, such as cadaver or porcine models.