CRII:RI: Methods for Dense Depth Estimation using Camera and mmWave Sensors

The ability to perceive the depth of the varying aspects of the environment is crucial to interacting with the world. An approximation of the distance of the hoop is what allows a basketball player to aim the shoot. Equipping computers with this ability can expand their perception of the world, enhance human-computer interaction, and advance emerging technologies like virtual reality/augmented reality. Although depth estimation as a problem has been studied for decades, there is still a need for better, cheaper, practical, and alternative approaches to meet the needs of many applications that depend on depth information. This project addresses this gap by building a depth estimation system that utilizes low-cost, ubiquitous wireless sensing devices operating at millimeter wave frequencies and cameras. By building systems of such ubiquitous sensors, this project enables device reuse, eases deployment efforts, and allows inexpensive, convenient alternative solutions to emerge. The proposed research can reduce the cost of future technologies like immersive telepresence, telehealth, etc., and make them accessible to technologically underrepresented communities. The project integrates research with education and inspires students from underrepresented communities to participate in research and education in computing. This project develops a depth estimation system using wireless sensors and cameras. First, a large-scale human activity and gesture recognition dataset will be collected using multiple cameras and wireless sensors. The dataset will serve as a benchmark to study the fundamental research challenges in multi-sensor solutions. Second, the project will develop methods to align the data obtained through different sensors. The methods will address two challenges: (i) how to align the data obtained through different sensors, such as cameras and wireless sensors, and (ii) how to align the data obtained through different perspectives from similar sensors. Third, the project will develop data-driven methods to estimate depth as a function of the pre-processed and aligned data obtained from the sensors. Finally, the project will investigate approaches that address the sparse and noisy signal measurements obtained from wireless sensors. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.