Semi-Supervised Building Extraction with Optical Flow Correction Based on Satellite Video Data in a Tsunami-Induced Disaster Scene

Huijiao Qiao, Weiqi Qian, Haifeng Hu, Xingbo Huang, Jiequn LiSensors 2024 · DOI

TL;DR

This paper introduces a novel semi-supervised method for extracting buildings from satellite video data in disaster scenarios. It combines an initial building prediction with an automatic correction module that leverages optical flow to refine segmentation results, significantly improving accuracy and reducing reliance on extensive manual labeling, which is crucial for rapid disaster response.

What Problem Does This Paper Address?

Accurate and timely building extraction is vital for disaster monitoring, emergency response, and damage assessment. While deep learning has advanced building extraction, existing methods face challenges:

  • Data Scarcity: Training robust models requires large, manually labeled datasets, which are expensive and time-consuming to create, especially in disaster-stricken areas where rapid assessment is critical.
  • Complex Disaster Scenes: Natural disasters, like tsunamis, drastically alter building appearances and create complex scenes with debris, rubble, and partial destruction, making traditional and even some advanced AI methods struggle with accuracy.
  • Dynamic Nature of Disasters: Satellite video data offers a dynamic view of disaster evolution, but extracting consistent building information from these changing scenes is difficult.

This paper aims to bridge this gap by developing a method that can achieve high-accuracy building extraction from satellite video data with minimal manual annotation, specifically addressing the challenges posed by dynamic and complex tsunami-induced disaster environments.

Methods

The proposed method consists of two main modules:

  1. Prediction Module:
    • Data Preprocessing: Satellite videos are converted into multi-frame images, and key frames are extracted and clipped to reduce computational load.
  2. Initial Building Extraction: A standard semantic segmentation network, SegNet, is used. To overcome the data scarcity issue, a limited set of manually labeled key frames (representing early, mid, and late disaster phases) are used to train SegNet. Data augmentation techniques are employed to expand this small dataset. This module provides an initial, potentially error-prone, building segmentation.
  3. Automatic Correction Module:
    • Identification of Erroneous Pixels: This module analyzes the initial segmentation results from adjacent key frames. It identifies discrepancies by comparing building area changes (using pixel counts) between frames. Two main types of errors are considered: decreasing building area (due to destruction) and increasing building area (due to debris accumulation). Specific criteria based on area differences and trends across sequential frames are used to pinpoint erroneous pixels.
  4. Optical Flow-Based Correction: For the identified erroneous pixels, the module uses optical flow to estimate the motion between adjacent frames. The OFATS (Optical Flow-based Adaptive Thresholding Segmentation) algorithm is employed to calculate dense optical flow fields. This motion information is then used to transfer accurate semantic information from a corrected adjacent frame to the erroneous pixels in the current frame, effectively refining the segmentation. This process is iteratively applied to small patches within the key frames.

The novelty lies in the semi-supervised approach that leverages limited labeled data and the automatic correction module that uses optical flow to dynamically refine segmentation based on inter-frame motion, rather than relying on more labeled data or complex model architectures.

Datasets

  • Indonesia Tsunami Video Data: The primary dataset used is a synthetic video captured by Digital Globe's WorldView satellite, documenting the aftermath of a tsunami in Petobo, Indonesia, on September 28, 2018. This video, after processing, resulted in 301 sequence frames.
  • Key Frames for Training: 21 key frames were extracted from the video, representing early, mid, and late stages of the disaster.
  • Ground Truth: Supervised classification results (using SVM) on these key frames were used as ground truth for training the initial SegNet model.
  • Data Augmentation: Extensive data augmentation techniques (rotation, scaling, flips, blur, gamma correction, noise injection, translation) were applied to generate approximately 20,000 augmented image pairs for training.

The paper does not introduce a new benchmark dataset but utilizes a specific disaster video for evaluation.

Key Insights and Ideas

  • Leveraging Video Dynamics for Correction: The core insight is that the temporal information in satellite video is not just for initial detection but can be actively used to correct errors in segmentation. Optical flow provides a powerful way to track pixel movement and infer accurate semantic labels.
  • Semi-Supervised Learning for Disasters: The paper demonstrates that a combination of a basic supervised model (SegNet) with a clever unsupervised correction mechanism can achieve high accuracy with significantly less manual labeling, which is a critical advantage in time-sensitive disaster scenarios.
  • Disaster Phases Matter: The method explicitly considers the different phases of a disaster (early, mid, late) by selecting key frames from each. This acknowledges that building damage and scene complexity evolve, and the correction strategy implicitly adapts to these changes.
  • Optical Flow as a Robust Feature: Using optical flow for correction is robust because it relies on pixel movement, which can still be detected even when building appearance is severely altered by damage or debris.

Key Results

  • Improved Accuracy: The proposed method significantly outperforms the baseline SegNet model. For example, in frame 179, building extraction accuracy improved from 79.15% to 85.15% after correction. In the more challenging mid-disaster phase (frame 339), accuracy increased from 75.45% to 84.39%. In the late phase (frame 439), accuracy improved by 2.49%.
  • Quantitative Metrics: The paper reports improved IoU, F1 score, and OA metrics across all three disaster phases compared to SegNet and other state-of-the-art methods like Swin-Unet, UNet++, PFNet, and SegFormer. For instance, in the late-phase frame, the proposed method achieved the highest IoU (72.15%), F1 score (78.99%), and OA (87.99%).
  • Computational Efficiency: While the correction module adds some computational overhead, the overall method is time-efficient. The total calculation time for all video data was 6.6 hours, significantly less than manual annotation time and competitive with other advanced models.
  • Robustness: The method shows robustness across different disaster phases and complex scenes, effectively handling fragmented and partially destroyed buildings.

Limitations and Future Work

  • "Cracks" in Correction: The paper acknowledges that in some areas, "cracks" appear in the corrected results. This is attributed to the nature of optical flow values, which are not always integers and require rounding, potentially leading to missing semantic information at pixel locations. The current solution involves assigning values from nearest pixels.
  • Damage Level Distinction: The current study extracts all building attributes as buildings, including intact, semi-destroyed, and destroyed ones. It does not distinguish between different levels of building damage, which is a critical metric for disaster evaluation.
  • Optical Flow Limitations: The OFATS algorithm, while effective, is a dense optical flow method that requires significant computational resources and can be limited by large motion amplitudes between corresponding pixels. If motion is too large, the algorithm may fail to accurately estimate optical flow, impacting correction precision.
  • Future Work Suggestions:
    • Develop the automatic correction module using sparse optical flow estimation to handle larger motion changes more efficiently.
  • Incorporate additional data sources (e.g., building damage levels) and fuse hand-crafted features with CNN-learned features to better distinguish damage levels.
  • Explore alternative methods for handling non-integer pixel positions resulting from optical flow calculations.

Summary updated

BibTeX
@article{qiao2024semi,
  author = {Huijiao Qiao and Weiqi Qian and Haifeng Hu and Xingbo Huang and Jiequn Li},
  title = {{Semi-Supervised Building Extraction with Optical Flow Correction Based on Satellite Video Data in a Tsunami-Induced Disaster Scene}},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24165205},
}
Figure