CCRI: Research Infrastructure: Planning-M: Multi-Modal Infrastructure for Enabling Social AI Research

Artificial intelligence (AI) for understanding social communication can play a key role in important application domains such as household robotics. However, current advancements in natural language processing and computer vision mainly focus on analyzing the surface structure of text and videos, lacking the ability to understand social communication deeply. To build large-scale AI systems that can better understand social interaction, this planning project aims at how to develop a process for collecting and annotating multi-modal and cross-cultural datasets of social communication in a responsible and sustainable way. The planning activities also include organizing workshops and panels around multi-modal social data collection to characterize the needs of this emerging social AI community. These planning activities will provide the foundation towards building the next generation of social AI models and ensuring that collection and use of the data complies with ethical and legal considerations. The resulting data infrastructure will facilitate future research on theory of mind, teamwork, and group decision-making. This planning project will demonstrate the potential of interdisciplinary research between natural language processing, computer vision, social science, law, and ethics. This project will be hosted via this website: socialai-data.org, where data, code, and data use protocols will be actively updated and maintained during the project period. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.