CAREER: Towards Proprioceptive Vision Through the Lens of Wearable Cameras

Understanding how humans interact with the physical world is essential for teaching intelligent systems to perform complex tasks effectively. Everyday activities, such as grasping a coffee mug and taking a sip, often involve the seamless integration of visual perception and motor control, a process that current intelligent systems struggle to replicate. This project aims to bridge the gap by developing a new family of visual sensing and learning approaches to track hand motion and interpret daily interactions, all using wearable cameras. Further, this project will demonstrate a key application in occupational safety and health by adapting the technology to assess injury risks in the workplace. By advancing the sensing and analysis of human interactions, this research will deepen the understanding of human intelligent behavior, drive the development of more capable intelligent systems, and expand practical applications of such systems. The project’s outputs, including open-source algorithms and hardware platforms, will be disseminated through public competitions, online courses, and diverse outreach activities. This project will advocate a new paradigm of proprioceptive vision, where an intelligent system integrates physical awareness with visual perception to understand how its actions relate to its observations. The project’s goal will be achieved through two interconnected research thrusts. Thrust 1 will develop computer vision approaches to harness wrist-mounted, spatially distributed, miniature single-photon cameras for reconstructing hands and in-hand objects. This will create a new solution for wearable hand tracking, supporting long-term recording under various lighting conditions. Thrust 2 will create machine learning techniques to integrate egocentric videos and hand motion for understanding hand-object interactions. This will advance the foundational representation of objects and actions and enable compositional reasoning of interactions. Finally, the outcomes from both thrusts will be integrated to address a significant health challenge: assessing the injury risk of mobile workers engaged in repetitive manual picking tasks in workplaces. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.