Gen-AI Airway Simulator for 3D Endoscopy

ABSTRACT Endoscopy plays a vital role in the diagnosis and treatment of various medical conditions and is regularly performed millions of times every year. However, endoscopy is both challenging to perform and often time- consuming to analyze from the resultant videos. Thus researchers have focused on developing many assistive and intelligent techniques to aid physicians in conducting endoscopies and to analyze the data. These techniques all share a common fundamental problem of 3D perception, i.e. recovering the 3D structure of the surveyed organ and localizing the endoscope in it from the captured 2D videos only. However current 3D perception techniques perform poorly on real clinical data. One key reason for failure is the lack of high-quality training data with 3D supervision for training 3D perception systems with Machine Learning algorithms. Current 3D perception systems rely on simple virtual models that cannot model complex geometric and reflective properties of internal organs, e.g. airway. In this proposal, we focus on developing a generative AI-based phantom airway model that can generate arbitrary geometric shapes and reflective properties of the mucus-layered airway surface conditioned on covariates such as age, sex, weight, and abnormalities. We aim to learn this generative airway phantom model from n=300 CT scans and n=300 endoscopy videos of different patients captured at UNC Chapel Hill. We first focus on designing a novel neural architecture that can generate realistic shapes from global and local latent codes with covariates, and a novel loss function that trains this generative model from CT scans and endoscopy videos (Aim 1). We then develop a novel strategy that can render realistic endoscopy videos by optimizing the material reflectance property modeling the mucus layered airway surface (Aim 2). We will demonstrate the effectiveness of our proposed generative-AI airway simulator by training state-of-the-art 3D perception systems on our simulator and existing virtual airway model BronchoPose dataset, and testing these 3D perception models on n=30 paired endoscopy videos with associated 3D CT scans. In summary, our proposed generative-AI-based airway simulator aims to provide high-quality realistic training data for various 3D perception systems. This will help in significantly improving various assistive and intelligent technologies, e.g. semi-autonomous navigation, 3D visualization, guidance to unsurveyed regions, automatic extraction of geometric properties, etc., for conducting and analyzing endoscopy and will help in incorporating these technologies into successful clinical practices. While we focus on the airway, our approach is general and can be used for building simulators of other organs of interest for endoscopy, e.g. GI tract.