III: A Declarative Data Management System for Semantic Multimodal Workflows

The amount of data stored as videos, audio recordings, and documents is growing rapidly and now far exceeds traditional structured data. Extracting useful knowledge from these diverse sources often requires combining information across formats. For example, a traffic engineer may need to analyze camera feeds in combination with road network data to form an accurate understanding of the situation. Today, performing such analyses requires significant programming expertise, is error-prone, and can be prohibitively expensive due to the cost of running artificial intelligence models on large volumes of data. This project develops a new open-source software system that allows users to express complex analyses over videos and documents using a simple, high-level interface, while the system automatically finds efficient ways to execute them. This project will advance the science of data management and by making large-scale multimodal data analysis accessible to a broader range of users, including journalists, city officials, and transportation planners. The resulting software and benchmarks will be released publicly to support further research and education. The project also supports workforce development through new course materials at the intersection of databases and artificial intelligence, and through training of graduate students and postdoctoral researchers. Specifically, this project builds a declarative data management system for authoring, optimizing, and executing semantic workflows that span multiple unstructured data modalities, focusing initially on documents and videos. The project designs a unified data model and domain-specific language (DSL) with composable operators that operate uniformly over document collections and video frame sequences, accompanied by a low-code interface for iterative workflow development. Building on this foundation, the project develops novel optimization techniques that transfer strategies across modalities. The system will be evaluated on real-world multimodal workloads drawn from collaborations in records analysis, freeway traffic monitoring, and city council meeting analysis, using benchmarks that measure execution time, inference cost, scalability, and output quality. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.