Using Multimodal Data and AI to Dynamically Map Flood Risk

Lydia Bryan-SmithAAAI 2022 · DOI

Introduction

Traditional flood warning systems often struggle with the trade-off between accuracy and latency. Currently, emergency services rely on two primary methods: static sensor networks, such as river gauges, which are geographically sparse, and physics-based hydrodynamic simulations. While these simulations provide high-fidelity results, their computational intensity makes them slow to run. Consequently, by the time a simulation completes, the environmental conditions on the ground may have already shifted, rendering the predictions outdated.

Furthermore, existing AI-driven solutions frequently suffer from "blind spots" because they rely on a single data modality. If a specific sensor fails or a data stream becomes noisy—such as atmospheric interference affecting radar—the entire prediction pipeline can fail. This research proposes a multimodal AI framework designed to move flood risk mapping into the realm of real-time "nowcasting" by fusing diverse data streams to ensure robustness and responsiveness.

Methodology

The proposed system architecture is designed to handle heterogeneous data through a three-stage processing pipeline. By integrating physical environmental data with real-time human observations, the model aims to provide a comprehensive view of evolving crises.

Figure 1: Multimodal AI framework for dynamic flood risk mapping.

1. Rainfall-to-Depth Autoencoder

The core predictive engine utilizes a Convolutional Neural Network (CNN) autoencoder, inspired by the U-Net architecture. This model processes two primary 2D inputs: rainfall radar data and topographic heightmaps. The goal is to learn the spatial relationship between precipitation and terrain to predict where water will accumulate.

2. Social Media Sentiment Analysis

To incorporate the human element of a disaster, the framework employs a Transformer encoder to analyze social media data. By filtering tweets via specific hashtags (e.g., #flashfloods, #StormChristoph), the model performs sentiment analysis to infer flood severity and humanitarian needs. This treats the public as a distributed network of "human sensors" capable of providing qualitative context that traditional sensors might miss.

3. Multimodal Fusion via CLIP

The final stage involves tying these disparate sources together using CLIP (Contrastive Language-Image Pre-training). CLIP is used as a base model to pair "nowcasted" water depth predictions with social media posts and images, helping to identify communities at the highest immediate risk.

Data Sources and Synthetic Labeling

Training a model for flood prediction is challenged by the scarcity of high-resolution "ground truth" water depth maps. To overcome this, the project utilizes a combination of real-world data and synthetic labels:

  • Environmental Data: Rainfall radar data is sourced from the UK Met Office, while topography heightmaps are provided by the Ordnance Survey.
  • Social Data: Real-time and historical tweets are scraped using event-specific hashtags.
  • Synthetic Labels: The AI is trained to "mimic" the output of HAIL-CAESAR, a high-performance hydrodynamic model. By using the physics model as a "headless" data generator, the AI learns underlying fluid dynamics patterns, allowing it to produce similar results at a fraction of the computational cost during an actual emergency.

Preliminary Results and Discussion

As this work represents a student abstract and work-in-progress, initial results are preliminary but promising. The project successfully established a pipeline for generating binary "water/no water" labels using the HAIL-CAESAR simulation and implemented a functional Transformer encoder for sentiment categorization.

However, the author noted "mixed results" during the initial training phases, particularly regarding the complexity of parsing rainfall radar data. A significant insight from this stage is the potential for multimodal resilience: the central hypothesis remains that a system drawing from multiple sources—radar, topography, and social media—will be more resilient to the failure or noise of any single modality.

Limitations and Future Directions

The current iteration of the model is limited to binary outputs, indicating only the presence or absence of water. Future work will focus on:

  • Granular Depth Prediction: Moving toward "binned" water depths (e.g., 0-50cm, 50cm-1m) to provide more actionable intelligence for rescue operations.
  • Expanded Modalities: Integrating satellite imagery to further refine the accuracy of flood maps.
  • System Integration: Fully unifying the CLIP-based fusion model into a single dashboard for real-time situational awareness.

Summary updated

BibTeX
@inproceedings{bryansmith2022using,
  author = {Lydia Bryan-Smith},
  title = {{Using Multimodal Data and AI to Dynamically Map Flood Risk}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21574},
}
Figure