FRR: Semi-Structured, Under-Specified, Partially-Observable Robotic Rearrangement

The project aims to develop advanced technologies for intelligent robots to efficiently and autonomously interact with objects in everyday, human environments, such as homes and grocery stores, given general, natural language task descriptions. This technology addresses significant societal issues, including the support of older adults in independent living. As people age, reduced mobility often leads to frequent and severe injuries due to impaired vision, home hazards, and weakness. Household robots can assist with tasks like retrieving, transferring, and rearranging items, such as setting up a dinner table or grabbing a jar from the back of a cabinet. Similarly, rearranging robots can assist with labor-intensive, repetitive inventory management tasks in retail operations. Such tasks, like tidying and restocking shelves, are labor-intensive and can lead to injuries, while these jobs are often difficult to fill and have high turnover rates. Reliably performing these object manipulation tasks in human, semi-structured environments involves significant uncertainty and remains challenging for modern robotics. Furthermore, new objects are frequently introduced and manipulated in semi-structured environments, such as modern homes or grocery stores, further complicating the task for robots. In particular, autonomous robots face multiple hurdles in solving manipulation tasks in these scenarios, including (1) a robot must derive a complete manipulation plan from implicit task specifications given by non-expert humans, (2) the robot must achieve accurate scene understanding in environments where prior knowledge of objects is not always available, and (3) the planning process must respect realistic partial observability constraints, where sensors like RGB-D cameras can only inspect portions of a scene at a time. To address the limitations of the state-of-the-art, the project will develop a novel Iterative Scene Understanding and Rearrangement Planning framework. The framework will build increasingly accurate models of a robot's environment progressively. The adaptive scene representation will contain the identities, geometries, and possible locations of partially observed objects, to a level sufficient for safely and effectively resolving human-assigned tasks. This representation will be leveraged to efficiently execute manipulation tasks provided by people as natural language commands under realistic visibility constraints. The project will also lay the groundwork for efficient implementations of this framework, aiming to deliver natural, high-quality solutions that achieve desirable guarantees, such as safety, resolution completeness, and solution optimality. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.