RI: Medium: Universal Deformable Shape Models with Varying Skeletal Structures

Understanding the three-dimensional (3D) structure of animals and humans from everyday images and videos is essential for a wide range of real-world applications - from analyzing animal motion in biological research to planning surgeries for children with variations in hand anatomy. This project supports the development of a new class of digital shape models capable of accurately representing deformable objects like animal bodies and human hands, even when their internal skeletal structures deviate from the norm. Unlike existing models that rely on a fixed skeleton, this project enables adaptive, learnable models that can accommodate diverse and atypical anatomies. By supporting flexible modeling across a broad spectrum of species and conditions, this work has the potential to advance research in biology, medicine, and education. The project includes plans for public release of tools and datasets, along with educational outreach involving students and domain experts. The research will develop a universal deformable shape modeling framework that integrates data from 3D scans, images, and videos to handle objects with varied skeletal topologies. The research includes three main thrusts: (1) learning mesh-based and implicit shape generators with disentangled latent representations for object type, shape, and pose; (2) constructing bone-driven shape priors that generalize to previously unseen skeletal structures, enabling modeling of rare or pathological forms; and (3) applying these models to inverse problems in 3D and 4D reconstruction from visual inputs. The project also introduces novel methods for keypoint detection, skeletal graph inference in open-vocabulary settings, and multimodal shape alignment, supporting robust reconstruction of deformable structures from visual cues. These capabilities will enable new applications, such as reconstructing animal morphologies for evolutionary studies and modeling hand deformities to assist surgical planning. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.