CAREER: Structure-Preserving Multimodal Alignment between Vision and Language

A grand challenge in artificial intelligence (AI) is to be able to process multimodal vision and language data, while preserving relationships across such modalities so that the linkages between the different modalities is sustained. Current machine learning systems do not fully grasp the structures and relationships that exist within human vision and language, and thus have difficulties producing the desired outcomes in terms of interpretability, efficiency, measurability, and causality. This project tackles the fundamental multimodal alignment problem in machine learning and will advance research in both computer vision and natural language processing, especially in the disruptive innovation areas of multimodal vision-language generation and understanding. It will lead to breakthroughs in both theoretical understanding as well as practical applications of vision and language. The techniques developed under this project could similarly be used to connect different types of latent structures across modalities and are not limited to vision and language. This would be extremely beneficial for responsible AI applications in the sciences, where people not only want to understand the relationship in data, but the structure and causal explanations. Such an understanding is also critical for reducing demographic biases that machine learning models exhibit. Through education, open-sourcing and outreach activities, this project will train and educate students of all levels - from K-12 to graduate - in AI, advance theoretical vision and language courses, reduce bias, and further democratize AI. Preserving structure is an essential component of understanding how to make machine learning models better and more reliable. This project aims to create novel and significant scientific advances in multimodal vision and language modeling with structure-preserving latent space alignment to build a bridge between vision and language. The project aims to increase the structural preserving nature for linguistic and visual embeddings and develop a map between the two latent representations that preserves the underlying structures. In particular, the project will achieve these goals through four thrusts: (I) Developing structure-preserving latent representations and mapping between vision and language; (II) Improving learning and data efficiency through latent structures; (III) Develop novel evaluation metrics through structural information to improve measurability; (IV) Develop a causal representation and interpretation framework. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.