CAREER: 4D Human-Object Interaction Understanding in the Wild

For many applications in robotics, social science, and medical analysis, it is critical to have computers to understand how humans interact with the physical world. The goal of this project is to build an artificial perception system that goes beyond 2D visual understanding and focuses on parsing the time dynamics and 3D structure of human-object interaction. Thus, to allow real-world applications, the system for understanding 4D (3D + time) human-object interaction is designed to generalize to unconstrained environments of daily life. The developed technologies will enable household robots to predict, and assist humans and doctors to analyze the behaviors of the elderly for providing better care, etc. The education and outreach activities of the project will advance the theoretical courses in computer vision and robotics, and offer practical guidance for undergraduate, graduate, and K-12 students to study in this field. This research develops a 4D human-object interaction understanding system, with a focus on generalization to environments in the wild. Existing approaches are limited to operating within constrained labs or uncluttered scenes. This is mainly due to current computer vision methods heavily relying on detailed 3D/4D annotations, which are usually unavailable for unconstrained environments. To address this limitation, the project provides a framework that will (i) recognize objects from videos in an open-world setting using only text-supervision and self-supervision; (ii) reconstruct and estimate the human and object structures in 3D using minimum 3D supervision and manually defined priors; and (iii) perform 4D interaction and contact relation reasoning between humans and objects using a 4D Interaction Graph. Based on the research studies, the project will develop a 4D Vision Education Platform on course materials focusing on both the generalization aspect of computer vision and the applications in robotic tasks. This platform will be applied in undergraduate and graduate courses, and tutorials for K-12 students. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.