III: Web-Scale Hand-Object-Centric Video Dataset Sourcing and Aggregation with Video-Based Data Retrieval for Robot Learning

Robots have the potential to assist people in homes, hospitals, factories, and other everyday settings. However, they still struggle to manipulate objects reliably in diverse environments. Tasks such as making contact with items, understanding how those objects move, and carrying out multi-step tasks all provide significant challenges. A central barrier to training robots to perform more tasks efficiently, is the lack of data. The field of artificial intelligence has advanced by learning from large collections of text, images, and videos. Robots do not yet have access to comparably large and diverse demonstrations of physical interaction. As a result, many robotic systems work only under the narrow conditions in which they are trained, and fail when objects, settings, or tasks change. Current efforts to collect robot data often require people or robots to record demonstrations one episode at a time, making the process slow, expensive, and difficult to scale. At the same time, online videos contain vast records of people interacting with objects, such as opening containers, using tools, and cleaning spaces. This project develops software infrastructure that turns this underused source of real-world interaction experience into structured, searchable data for robot learning. The system will identify, filter, and organize relevant videos so that researchers can search for examples of particular object interactions. The resulting shared data resources, retrieval tools, and open infrastructure will lower the cost of building capable robot systems, support safer and more adaptable automation, and broaden access to robotics research. This project will create a hand-object-centric data engine for sourcing, organizing, and retrieving web-scale video data for robot learning. The work is organized around three main activities. First, the project will develop a pipeline that decomposes raw video pixels into structured representations of hand-object interactions, called MotionBits. Using kinematics-driven supervision, the representation will exploit causal agency and contact discontinuities to autonomously filter background noise and retain functionally relevant interactions. Second, the project will develop an autonomous sourcing and retrieval engine based on interaction-aware predictive learning. This engine will organize and retrieve videos based on the physical dynamics of interaction, rather than visual appearance alone, enabling retrieval by example videos as well as combined video and text queries that specify the action, object, or task context. Third, the project will establish an evolving database infrastructure that grows from existing research datasets and public web videos into a closed-loop open resource that expands and refines itself over time. The resulting infrastructure is expected to enable robots with different embodiments to learn robust manipulation policies from large-scale web video, providing a scalable foundation for the next generation of intelligent robots. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.