CAREER: Active Scene Understanding By and For Robot Manipulation

Despite significant progress, most robot perception systems today remain limited to "seeing what they are asked to see" – detecting pre-defined categories of objects by watching static images or videos. In contrast, humans constantly decide "what to see" and "how to see it" using active exploration. This ability is central to problem-solving and adaptability to novel scenarios but remains missing from robots today. To bridge this gap, this Faculty Early Career Development (CAREER) project aims to study a self-improving robot perception system using manipulation skills – referred to as active scene understanding. The framework suggested in this project improves a robot's fundamental capabilities in perception and planning and therefore impacts many application domains such as service robots or field exploration, where robots need to rapidly analyze their environments in order to swiftly react to evolving situations. The research and education plans are integrated through a Cloud-Enabled Robot Learning Platform, which allows students to participate in robotics education and research without the limits of robot and compute hardware accessibility. This project tackles a number of challenges in active scene understanding to achieve a unified and practical framework. The key idea of the approach is to leverage the synergies between a robot's perception and interaction algorithms to create self-supervisory signals. On the one hand, the robot can use its own actions and the corresponding action effects (i.e., visual observation of subsequent states) as ground truth labels for training its visual predictive model. On the other hand, the robot can also use the statistics provided by the perception model (e.g., uncertainty, novelty, and predictability) as a reward signal to improve its manipulation policy. Ultimately, the robot could combine the learned visual predictive model and manipulation policy to facilitate efficient action planning for downstream tasks. This project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE). This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.