Collaborative Research: CPS: Medium: Robotic Perception and Manipulation via Full-Spectral Wireless Sensing

Robotic manipulation and automation systems have received a lot of attention in the past few years and have demonstrated promising performance in various applications spanning smart manufacturing, remote surgery, and home automation. These advances have been partly due to advanced perception capabilities (using vision and haptics) and new learning models and algorithms for manipulation and control. However, state-of-the-art cyber-physical systems remain limited in their sensing and perception to a direct line of sight and direct contact with the objects they need to perceive. The goal of this project is to design, build, and evaluate a cyber-physical system that can sense, perceive, learn, and manipulate far beyond what is feasible using existing systems. To do so, the research will explore the terahertz band, which offers a new sensing dimension by inferring the inherent material properties of objects via wireless terahertz signals and without direct contact. This project will also explore radio-frequency signals that can traverse occlusions. Building on these emerging sensing modalities, the core of the project focuses on developing full-spectrum perception, control, learning, and manipulation tasks. The success of this project will result in CPS system architectures with unprecedented capabilities, enabling fundamentally new opportunities to make robotic manipulation more efficient and allowing robots to perform new complex tasks that have not been possible before. The project will enable robotic perception via full-spectral wireless sensing in order to unlock unprecedented robotic manipulation capabilities. This research involves learning synergies between sensing and control- whereby sensing is used for control and vice-versa - to optimize the end-to-end cyber-physical tasks. In particular, this research includes three inter-connected thrusts: (i) It will enable a new sensing modality that exploits high-resolution terahertz frequencies for robotic imaging and inference; (ii) It aims to build a new learning platform for full spectrum (mmWave, THz, and vision) perception to enable beyond-vision perception and reasoning in non-line-of-sight and cluttered environments, where optical systems lack in performance; and (iii) It presents a platform to learn the synergies between sensing and control to further co-optimize the end-to-end robotic manipulations tasks. These capabilities can open up entirely new realms of possibility to industrial robotics as well as assistive, warehousing, and smart home robotic. The research will be evaluated through extensive experimentation, prototype design, and system implementation. The results will be disseminated through close collaboration with industry and publications in top research venues. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.