CRII: RI: Modeling and Understanding the Invisible World in Thermal Modality

Most current scene modeling and understanding algorithms target at visible scenes with sufficient light supplied and the exposed objects. However, it is common for situations to occur in which there is little light or objects are hidden, such as nighttime in outdoor or indoor environments when the power outage happens, water press of plants, and hollows in the bridge deck. Thermal images capture the long-wave infrared signals to generate images and have broad applications in transportation, agriculture, and surveillance. This project develops 3D modeling and understanding methods based on thermal images. The developed 3D modeling technology can enable autonomous driving vehicles to detect the scene depth from thermal images to avoid traffic accidents. With a 3D thermal model, hidden gas pipes in buildings and cracks in bridges can also be inspected more reliably. Specifically, this project addresses the 3D thermal sensing and understanding issues through the following thrusts: (1) thermal image feature detection and extraction, (2) translation between thermal and visible images, (3) unsupervised 3D thermal reconstruction methods based on both image sequence and a single image, and (4) 3D thermal model understanding, particularly for nighttime and invisible objects. The algorithms developed in this project will enable 3D reconstruction and understanding tasks for invisible scenes. The output from this project will allow related services provided for visible objects and during the daytime to be also available for invisible objects and during nighttime, such as hidden object inspection and navigation in the nighttime. The enhancement of invisible scene processing capability will improve safety and cybersecurity across different scenarios, from daytime to nighttime and from exposed objects to invisible object properties. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.