Collaborative Research: RI: Small: Shape Configuration Learning and Its Applications

This project aims to enhance how computers understand and recognize shapes in images and 3D data. While current advanced computer systems are powerful, they struggle to perceive objects as humans do—as whole shapes made up of interconnected parts. This project seeks to teach computers to focus on the critical parts of an image that constitute an object, disregarding the background and other distractions. The project is also developing methods for computers to comprehend the relationships between different parts of an object, akin to recognizing a chair by identifying its seat, legs, and back in a specific arrangement. This will be achieved within recent deep learning frameworks without explicitly referencing such parts. The improved visual perception capabilities could help computers better to perform tasks like object recognition, 3D position estimation, and understanding shapes from various viewpoints. The potential applications of this research are vast, including advancements in medical imaging, dental radiology, augmented reality, assisted driving technologies, and robotics. Additionally, the project will involve students from high school to doctoral programs in research, with a focus on including underrepresented groups. The project creates a framework for deep learning models to learn configural (or holistic) shape representation and arrangements of object parts, demonstrating their applicability to various computer vision tasks. It will augment the learning framework of vision transformers to learn how to restrict the attention of image patches. This is achieved by adding a new branch that computes an auxiliary loss called object-focused attention, which limits the attention to patches belonging to the same object. This allows transformers to gain a better understanding of configural object shapes by largely ignoring the background and other objects. Additionally, the project will develop a novel graph-transformer-based shape configuration framework named ShapeGT for generic shape understanding tasks. ShapeGT will include several new techniques and applications, including (1) novel between-edge attention and edge-to-node attention modules, (2) joint graph learning and matching algorithms, (3) view encoding techniques for multi-view analysis, and (4) cross-modal fusion mechanisms for capturing 3D-to-2D (shape-to-image) interactions. Due to the wide range of applications of shape analysis, the project is expected to have broader impacts on fields beyond computer vision, such as dental maxillofacial radiology, augmented reality, molecular biology, assistive driving, medical image analysis, and robotics. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.