CAREER: Deep Neural Networks That Can See Shape From Images: Models, Algorithms, and Applications

The past decade has witnessed the success of deep learning in image processing and computer vision. However, increasing evidence has shown that deep neural networks are strongly biased towards seeing image textures rather than geometric shapes. The goal of this project is to create a new family of deep networks that can analyze shapes from images and leverage this perspective to advance image analysis and machine learning with mathematical tools from geometry, statistics, and optimization. The technical component of this research will merge the largely disconnected scientific areas of geometric shape modeling, machine learning, and image analysis. The applied components of this research will have an immediate and long-lasting impact on a wide range of real-world applications, including quantitative analysis of magnetic resonance imaging for early neurodegenerative disease detection, computed tomography for anatomic pathology tracking, and satellite images for environmental monitoring. Moreover, the multidisciplinary nature of this research will provide unique opportunities to engage students and researchers from diverse backgrounds broadly across engineering, mathematics, and medicine. Current deep neural networks are incapable of quantifying and analyzing shapes presented in images, and their performances are limited by the high dimensionality of the training data. This research develops new algorithmic foundations that (i) equip deep neural networks with the functionality of learning shape representations and quantifying shape changes to best support image analysis; (ii) enable end-to-end approaches to transform raw image data into spaces of shape features that are easily and reliably compared across individuals, groups, or time sequences; and (iii) provide robust, scalable, and efficient inferences for training large-scale and high-dimensional image datasets. This research will not only expand the frontier of deep-learning-based image technologies but also profoundly inspire broader academic communities beyond image analysis and computer vision, such as computational anatomy and graphics. Results and tools produced in this research will be tightly integrated into educational activities and will be disseminated to general communities through open-source repositories, as well as tutorials in conjunction with conferences, seminars, workshops, and invited talks. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.