UNSUPERVISED HARMONIOUS IMAGE COMPOSITION FOR DISASTER VICTIM DETECTION

Nan Zhang, Francesco Nex, George Vosselman, Norman Kerle2022 · DOI

Abstract

Deep detection networks trained with a large amount of annotated data achieve high accuracy in detecting various objects. However, disaster victim detectors suffer from a lack of real-world training data, as images of victims partially buried by debris are difficult to obtain. This paper proposes generating realistic synthetic images for training. We extract human body parts from open-source datasets and paste them into ruins background images. An unsupervised generative adversarial network (GAN) then harmonizes these parts to fit the background's style (illumination, texture, and color). Fine-tuning a YOLOv5 model on these images significantly improves detection performance, increasing AP@[.5 : .95] from 19.5% to 35.7% and [email protected] from 32.4% to 59.5%.

Introduction

Object detection models have reached impressive levels of maturity for common categories like pedestrians and vehicles. However, in search-and-rescue scenarios following earthquakes or building collapses, these models often fail. Victims are frequently partially buried, with only limbs or torsos visible, and their appearance is obscured by the gray dust and debris of the disaster site. This "domain gap" between standard datasets (e.g., COCO) and real-world disaster scenes necessitates specialized training data that is currently scarce.

Methodology

The proposed framework addresses the data scarcity problem through a three-stage synthetic data generation and training pipeline.

1. Image Composition

Human body parts (limbs, torsos, or full bodies) are extracted from the LIP (Look Into Person) dataset using semantic masks. These parts are then randomly placed onto background images of earthquake ruins and collapsed structures.

2. Unsupervised Image Harmonization

A simple "cut-and-paste" approach results in visual inconsistencies that hinder model generalization. To bridge this gap, we employ an unsupervised GAN to harmonize the foreground body parts with the background environment.

Initial composite image showing a human limb pasted onto a debris background without harmonization.

The GAN architecture utilizes a U-Net generator with attention layers and dual discriminators (Global and Local). The Global Discriminator ensures overall image realism, while the Local Discriminator focuses on the boundary between the pasted limb and the debris. This process effectively simulates the "dust factor," adjusting the color and texture of the body parts to match the surrounding ruins.

Harmonized composite image where the limb's lighting and texture have been adjusted to match the dusty, gray environment of the ruins.

3. Model Fine-tuning

The final harmonized images are used to fine-tune a YOLOv5 detector, enabling it to recognize partially obscured human features in disaster-specific contexts.

Results and Discussion

The effectiveness of the harmonization process was evaluated using three versions of YOLOv5. The YOLOv5l (Large) model showed the most significant gains. While the baseline model pre-trained on COCO achieved only 19.5% AP@[.5:.95], fine-tuning on simple composite data raised this to 33.6%. The addition of the unsupervised harmonization step further boosted performance to 35.7% AP@[.5:.95] and 59.5% [email protected].

Key insights from the study include:

  • The Dust Effect: The GAN's ability to automatically apply "gray" tones to limbs mimics real-world conditions where victims are covered in debris dust.
  • Local Context: The local discriminator is vital for ensuring that small, partially visible body parts are not ignored by the network.
  • Limb Proxies: Training the model to detect body parts rather than whole persons is a more robust strategy for disaster scenarios.

Conclusion

This research demonstrates that unsupervised image harmonization can effectively synthesize high-quality training data for niche applications where real data is unavailable. By bridging the domain gap between standard person detection and disaster-specific environments, we can significantly enhance the capabilities of search-and-rescue technology. Future work will explore night-time detection using thermal sensors and further refine the model's ability to detect both partial and whole bodies.

Summary updated

Abstract

Deep detection networks trained with a large amount of annotated data achieve high accuracy in detecting various objects, such as pedestrians, cars, lanes, etc. These models have been deployed and used in many scenarios. A disaster victim detector is very useful when searching for victims who are partially buried by debris caused by earthquake or building collapse. However, considering that larger quantities of real images with buried victims are difficult to obtain for training, a deep detector model cannot give full play to its advantages. In this paper we generate realistic images for training a victim detector. We first randomly cut out human body parts from an open source human data set and paste them into the ruins background images. Then, we propose an unsupervised generative adversarial network (GAN) to harmonize the body parts to fit the style (illumination, texture and color characteristics) of the background. These generated images are finally used to fine-tune a detection network YOLOv5. We evaluate both the AP (average precision) for IoU (Intersection over Union) 0.5 and for IoU ∈ [0.5:0.05:0.95], which are denoted as AP@0:5 and AP@[.5 : .95], respectively. The best experimental results show that the YOLOv5l pre-trained on the COCO data set performs poorly on detecting victims, and the AP@[.5 : .95] is only 19.5%. The model that uses our composite images as fine-tuning data can effectively detect victims, and increases the AP@[.5 : .95] to 33.6%. The AP@0:5 increases from 32.4% to 53.4%. Our unsupervised harmonization method further improves the results by 2.1% and 6.1%, respectively.
BibTeX
@inproceedings{zhang2022unsupervised,
  author = {Nan Zhang and Francesco Nex and George Vosselman and Norman Kerle},
  title = {{UNSUPERVISED HARMONIOUS IMAGE COMPOSITION FOR DISASTER VICTIM DETECTION}},
  booktitle = {˜The},
  year = {2022},
  doi = {10.5194/isprs-archives-xliii-b3-2022-1189-2022},
}
Figure