Flood Event Analysis Base on Pose Estimation and Water-Related Scene Recognition

Khanh-An Quan-Chi, Tan-Cong Nguyen, Vinh-Tiep Nguyen, Minh-Triet TranMediaEval 2019 · OpenReview

Abstract

During flood disasters, distinguishing between minor wet streets and life-threatening flooding is crucial for emergency response. This paper presents a multi-stage computer vision pipeline developed for the MediaEval 2019 Challenge. By integrating pose estimation (OpenPose), instance segmentation (Mask R-CNN), and scene recognition, the system identifies dangerous flood levels by using the human body as a relative measurement tool. The approach achieved top performance in the Multimodal Flood Level Estimation (MFLE) task with an F1-score of 0.8831.

Introduction

Natural disasters like floods generate a massive volume of social media and news imagery, often overwhelming emergency responders. To prioritize rescue efforts, it is critical to automatically distinguish between "wet streets" and "dangerous flooding." This research addresses two specific tasks from the MediaEval 2019 Multimedia Satellite Task: Image-based News Topic Disambiguation (INTD), which ensures images match the reported flood context, and Multimodal Flood Level Estimation (MFLE), which identifies images where at least one person is standing in water at least knee-high.

Methodology

The authors proposed a four-stage pipeline for the MFLE task, transitioning from coarse scene understanding to fine-grained anatomical analysis.

The multi-stage pipeline for flood level estimation, combining scene recognition, person detection, and pose analysis.
  1. Water-Related Scene Recognition: A ResNet-50 model pre-trained on the Places365 dataset filters out images without water, acting as a high-recall gate.
  2. Person Detection: Faster R-CNN identifies images containing people; those without are discarded.
  3. Pose and Water Analysis: OpenPose extracts human keypoints (knees, hips, shoulders). Simultaneously, a custom WaterClassifier (ResNet-50) is trained on the bottom third of images to distinguish water from other surfaces.
  4. Paired Mask and Pose Prediction: Mask R-CNN provides segmentation. By calculating the Intersection over Union (IoU) between OpenPose bounding boxes and Mask R-CNN segments, the system "pairs" a person's pose with their visual mask. This allows the model to check if knee keypoints are submerged or if the thigh-to-body ratio indicates deep water.

Key Insights: The Human Body as a Yardstick

A core innovation of this work is using the human skeleton as a relative measurement tool rather than attempting to estimate absolute water depth.

Visual examples of pose estimation used to detect water levels relative to human anatomy.

If a "knee" keypoint is missing or submerged relative to the "hip," it serves as a strong proxy for flood depth. The authors also employed a heuristic "water-cropping" technique, training the WaterClassifier specifically on the bottom 1/3 of frames to reduce noise from sky or buildings. Furthermore, by pairing masks with poses, the model can handle edge cases like people on boats or swimming, distinguishing those "above" the water from those "in" it.

Results and Evaluation

The pipeline was evaluated on the MediaEval 2019 Challenge Dataset. The visual-only models significantly outperformed text-only models, which achieved F1-scores around 0.53.

  • MFLE Task: The best visual model reached an F1-Score of 0.8831, the highest in the challenge.
  • INTD Task: An EfficientNet-based approach achieved an F1-Score of 0.8850.

Conclusion

While highly effective, the system faces limitations when OpenPose fails to detect keypoints in deep-water swimming scenarios where only heads and shoulders are visible. Future work will focus on improving the robustness of the water-related scene classifier and refining the detection of partially submerged bodies where standard keypoints are occluded.

Summary updated

BibTeX
@inproceedings{quanchi2019flood,
  author = {Khanh-An Quan-Chi and Tan-Cong Nguyen and Vinh-Tiep Nguyen and Minh-Triet Tran},
  title = {{Flood Event Analysis Base on Pose Estimation and Water-Related Scene Recognition}},
  booktitle = {MediaEval},
  year = {2019},
  url = {https://openreview.net/forum?id=-z63FwryPQ},
}
Figure