Enabling Fast and Accurate Crowdsourced Annotation for Elevation-Aware Flood Extent Mapping

Landon Dyken, Saugat Adhikari, Pravin Poudel, Steve Petruzza, Da Yan, Will Usher, Sidharth Kumar

TL;DR

This paper introduces FloodTrace, a web-based tool that significantly speeds up and improves the accuracy of flood extent mapping by leveraging elevation data and crowdsourcing. By making advanced topological segmentation accessible online and integrating elevation-guided annotation, FloodTrace enables non-experts to contribute high-quality training data for machine learning models, achieving expert-level performance with a fraction of the effort.

What Problem Does This Paper Address?

Accurate flood extent mapping is crucial for disaster management, urban planning, and insurance. While deep learning models have shown promise, they require vast amounts of high-quality annotated training data. Manually labeling these large datasets, especially for high-resolution imagery, is extremely time-consuming and expensive, often requiring domain experts. Existing semi-automatic tools have limitations in terms of accessibility, efficiency, and integration with elevation data, which is vital for accurate flood modeling. This paper addresses the need for a more efficient, accessible, and accurate method for crowdsourcing flood extent annotations that can produce training data comparable to expert-labeled datasets.

Methods

FloodTrace is a web-based application that combines several key innovations:

  • Elevation-Guided Annotation: The core of FloodTrace is its use of Digital Elevation Model (DEM) data to guide annotation. This is implemented through two main tools:
  • Breadth-First Search (BFS) Tool: An adaptation of a previous method, this tool labels connected pixels based on elevation differences, allowing users to quickly mark flooded or dry areas by selecting a seed pixel.
  • Topological Segmentation Tool: This novel approach brings advanced topological data analysis (specifically contour trees and simplification by persistence) to the web. It segments elevation data into meaningful features at various scales, allowing users to label entire segments with a single click.
  • Interactive 3D Visualization: The application renders aerial imagery and DEM data as a 3D mesh, providing users with an intuitive way to explore the terrain and understand elevation relationships, which is crucial for accurate annotation.
  • Crowdsourcing Framework: FloodTrace is designed for easy access by non-experts via a web interface. It includes features for aggregating annotations from multiple users and visualizing areas of uncertainty or disagreement.
  • Uncertainty Visualization and Correction: Researchers can use aggregate views and variance maps to identify and correct inaccuracies in crowdsourced annotations before using them to train models.

Datasets

  • Hurricane Matthew Imagery: High-resolution aerial imagery from Hurricane Matthew in North Carolina was used for the user study.
  • Digital Elevation Model (DEM) Data: Corresponding DEM data from the University of North Carolina Libraries was used to inform the elevation-guided annotation tools.
  • Newly Introduced Benchmark: The paper introduces a dataset of 1,321 unique annotations collected from 266 graduate students across eight distinct regions, covering over 3.5 billion pixels. This dataset, along with metadata, is open-sourced.

Key Insights and Ideas

  • Elevation is Key for Accuracy: Integrating elevation data into the annotation process significantly improves accuracy, even for untrained users, by providing physical constraints that guide labeling.
  • Topological Segmentation on the Web is a Game-Changer: Bringing complex topological analysis tools to a web-based platform democratizes their use, enabling efficient segmentation and annotation that was previously inaccessible to most researchers.
  • Crowdsourcing Can Match Expert Quality: With proper aggregation and correction mechanisms, crowdsourced annotations can yield training data that is as effective as data labeled by domain experts, but at a fraction of the cost and time.
  • User Preference vs. Efficiency: While users might gravitate towards familiar tools like brushes (algorithm aversion), elevation-guided and segmentation tools offer substantial gains in both accuracy and speed, highlighting the importance of effective tool design and user education.
  • Soft Labels Improve Model Performance: Training deep learning models with "soft labels" (probabilities derived from aggregated annotations) rather than hard binary labels leads to better performance, suggesting that encoding uncertainty is beneficial.

Key Results

  • Efficiency Boost: The median annotation time for users employing the segmentation tool was significantly lower, at 17.45 minutes, compared to the brush tool (39.97 minutes). Elevation-guided tools generally reduced annotation time by more than half compared to the state-of-the-art.
  • Accuracy Improvement: Elevation-guided tools (segmentation and BFS) resulted in more accurate annotations compared to the manual brush tool.
  • Expert-Level Model Performance: Machine learning models trained on FloodTrace's corrected crowdsourced annotations achieved accuracy and F-scores comparable to models trained on fully expert-created reference labels (94.4% vs. 94.2% accuracy).
  • Time Savings: Correcting crowdsourced annotations took an average of 24 minutes per region, compared to 116 minutes for creating reference annotations, demonstrating a substantial workflow improvement.
  • User Study Scale: The study involved 266 graduate students who contributed a total of 1,321 unique annotations.

Limitations and Future Work

  • Two-Class Limitation: FloodTrace currently only supports labeling for "flooded" and "dry" classes. Future work could extend this to more specific labels like "permanent water bodies."
  • DEM Limitations: The use of DEM data doesn't account for above-ground structures (e.g., buildings) that can influence flood propagation. Incorporating Digital Surface Models (DSM) or LiDAR data could address this.
  • Data Synchronization: The system assumes elevation data and imagery are collected at similar times. Deviations could lead to errors.
  • Algorithm Aversion: Users may still prefer simpler tools. Future work could explore ways to enforce or encourage the use of more efficient, elevation-guided tools.
  • Urban Environments: The current approach might struggle in heavily urbanized areas where above-ground structures significantly alter flood dynamics.

Summary updated

Abstract

Mapping the extent of flood events is a necessary and important aspect of disaster management. In recent years, deep learning methods have evolved as an effective tool to quickly label high-resolution imagery and provide necessary flood extent mappings. These methods, though, require large amounts of annotated training data to create models that are accurate and robust to new flooded imagery. In this work, we present FloodTrace, a web-based application that enables effective crowdsourcing of flooded region annotation for machine learning applications. To create this application, we conducted extensive interviews with domain experts to produce a set of formal requirements. Our work brings topological segmentation tools to the web and greatly improves annotation efficiency compared to the state-of-the-art. The user-friendliness of our solution allows researchers to outsource annotations to non-experts and utilize them to produce training data with equal quality to fully expert-labeled data. We conducted a user study to confirm our application’s effectiveness in which 266 graduate students annotated high-resolution aerial imagery from Hurricane Matthew in North Carolina. Experimental results show the efficiency benefits of our application for untrained users, with median annotation time less than half the state-of-the-art annotation method. In addition, using our application’s aggregation and correction framework, flood detection models trained on crowdsourced annotations were able to achieve performance equal to models trained on fully expert-labeled annotations, while requiring a fraction of the expert’s time.
BibTeX
@inproceedings{dyken2025enabling,
  author = {Landon Dyken and Saugat Adhikari and Pravin Poudel and Steve Petruzza and Da Yan and Will Usher and Sidharth Kumar},
  title = {{Enabling Fast and Accurate Crowdsourced Annotation for Elevation-Aware Flood Extent Mapping}},
  booktitle = {PacificVis},
  year = {2025},
  doi = {10.1109/pacificvis64226.2025.00025},
}
Figure