ATD: Spatiotemporal Foundation Models for Multimodal Threat Detection at Scale

Foundation models have led to a paradigm shift in the areas of Natural Language Processing (NLP) and Computer Vision (CV). The result is that high-dimensional complex data from those domains can be used directly to realize machine learning (ML) and data science capabilities with very small amounts of labeled data, method development, and effort. This project will realize these benefits for threat detection and large-scale analysis of spatiotemporal data. Doing so will unlock the massive potential of existing data collection efforts, allowing for the fusion of disparate datasets, deeper understanding of complex threat dynamics, and enabling the rapid deployment of plausible analytic solutions to comprehend and handle various types of threats. This project will advance the AI/ML research community’s knowledge of the full range of capabilities of these large multi-task modeling strategies as well as contribute methods for scaling the proposed modeling strategies to very large, complex, multimodal datasets. Finally, the work will also advance knowledge of the dynamics and processes underpinning the evolution of threat in the demonstration applications. If successful, the project will enable new capabilities in urban planning, public safety, and environmental monitoring. It will also inspire and shape the next generation of data scientists and artificial intelligence experts through research mentoring and dissemination of results. This work will develop new foundation models and eventually generative Artificial Intelligence (AI) capabilities for spatiotemporal data, and is structured as three complementary research thrusts. The first will develop multivariate data representations for spatiotemporal processes. The second will focus on scaling models to very large datasets, by exploring specialized neural network architectures and developing highly expressive graph networks with temporal aspects, incorporating irregular multimodal spatiotemporal data. The third will develop methods for integrating and regularizing large models with existing highly scalable spatial detection algorithms such as fast subset-scanning. This will require ensuring that the latent spatiotemporal representations contain correct information and controlling the types of information extracted therefrom. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.