Career: Learning Multimodal Representations of the Physical World

Touch and hearing convey physical properties about the world that are difficult to perceive from vision alone. The objective of this project is to give machine perception systems the ability to form cross-modal associations between these three sensory modalities, such as the ability to predict how an object will feel or sound from sight. These cross-modal associations can also be obtained directly via sensors, making them well-suited to creating autonomous systems that learn to physically interact with the world without human-provided supervision. The project's integrated education and outreach activities will also advance an understanding of multimodal machine learning for a general audience, and for students at multiple levels. This project aims to learn material properties and microgeometry through cross-modal associations between sight, sound, and touch. It does this through four research thrusts. First, it aims to capture 3D multimodal representations by registering observations from all modalities into a unified 3D model, using estimated visual geometry to obtain dense estimates of touch and sound from sparse observations. Second, it aims to generate space-time reconstructions of objects from touch and sound during physical interaction, using cross-modal visual supervision. Third, it aims to learn material representations that capture acoustic properties, as well as methods that integrate these representations into 3D sound synthesis models. Finally, it aims to simulate and learn physical interactions within captured 3D multimodal scenes. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.