Multimodal Disaster Impact Assessment Models for Enhanced Resilience

Immediately after a major hazard event (e.g., wildfire, earthquake, flood), a prompt assessment of the geographic distribution and severity of infrastructure damage is vital to the success of the emergency response and early recovery planning. This situational awareness is an important part of the decision-making processes that are implemented by facility owners, users, emergency responders and local and state officials. Conversely, a general lack of knowledge about the impacted state of the built environment can lead to a disorganized public response and slower recovery. While a comprehensive assessment of the extent and distribution of infrastructure damage can be obtained from in-person inspections conducted by building professionals, depending on the scale of the event, this can be a lengthy, resource intensive process. This Disaster Resilience Research Grants (DRRG) project will address this challenge by utilizing principles from artificial intelligence (AI) to develop near real-time infrastructure damage prediction models that can process and utilize different types of data and information (e.g., images, text, tabular data). By advancing our ability to effectively integrate disparate information sources, this project aims to transform the way that physical damage to infrastructure is estimated in the aftermath of a major disaster event, thereby enhancing the emergency response and recovery planning phases that follow. The research will provide training for doctoral and masters students and an opportunity to teach undergraduates from different backgrounds how science and engineering coupled with AI technologies can be used to improve community response to extreme events. Fundamental concepts and methodological advancements in multimodal learning will be used to transform and enhance infrastructure damage prediction models for use in the immediate post-event environment. The state-of-the-art in image-based damage assessment will be advanced along two dimensions: (1) developing Vision Transformer-based methods and (2) establishing a self-supervised learning methodology for training the models using large collections of unlabeled data. A new knowledge base will be established around the broad area of multimodal data fusion for infrastructure damage prediction models. Specific questions regarding unified representation, translation across and alignment between modalities and data fusion will be answered. A new type of hazard-agnostic infrastructure damage prediction model will also emerge from this research. Such a model will have the ability to receive an integrated representation of one or more types of input modalities (i.e., image, text, and engineering/tabular data) and produce, as output, an infrastructure damage level that is agnostic to the type of causal event (e.g., earthquake or hurricane). Using a comprehensive data set from multiple natural hazard events (hurricane and earthquake), the project will include experiments to shed new light on the ability of both hazard-specific and hazard-agnostic multimodal models to enhance early-stage infrastructure damage assessments for increased situational awareness and enhanced resilience. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.