OAC Core: Cyberinfrastructure for Multi-Stream Architectures Applied to Computer Vision: Efficiency via Co-Design of Network Architectures and Framework Operators

Many complex tasks are hard because they make use of multiple kinds (modes) of data at once, such as replying to a question based on both the question-asker’s words and gestures, or generating a video based on a screenplay, a directing style, and knowledge about the audience. Many artificial intelligence approaches have been attempted on these multimodal tasks. While conventional deep neural networks struggle with them, multi-stream architectures are an emerging approach that has been shown to perform better. However, despite their potential, the broader adoption and advancement of multi-stream methods and models are limited by gaps in existing algorithms and their implementations, as well as a lack of foundational design knowledge. This project aims to address these shortcomings by integrating modern developments in deep learning with efficient algorithmic implementations to produce a library of state-of-the-art multi-stream models. The anticipated outcomes include: (1) a suite of high-performance multi-stream foundation models with applications to object detection, text-based image segmentation, and audio-video analysis; (2) optimized algorithms that enhance the efficiency of their internal mechanisms; (3) and new foundational knowledge to guide the design of future multi-stream systems. These outcomes are expected to advance the scientific frontier of deep learning while simultaneously supporting the community with cyberinfrastructure that includes publicly available models and high-quality software implementations. This project will explore two major directions that will advance cyberinfrastructure for multi-stream architectures: (1) the design, training, and scaling of novel multi-stream architectures, and (2) the development of efficient implementations for their core components based on hardware-software co-design. Combining these approaches, we will synthesize theoretical and algorithmic improvements with efficient implementations, thus providing robust cyberinfrastructure for future work. Evaluations will be conducted across diverse tasks, including classification and generative modeling, and on a range of hardware platforms, from edge devices to desktop-grade systems and high-performance computing systems. Comparisons with existing state-of-the-art neural networks will enable us to quantify improvements in both accuracy and efficiency, with particular attention to expanding the accuracy-efficiency Pareto frontier. All datasets, pretrained models, and optimized implementations, such as custom kernels and operators, will be made publicly available to maximize impact and reproducibility. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.