DEAL: Difficulty-Aware Active Learning for Semantic Segmentation

Shuai Xie, Zunlei Feng, Ying Chen, Songtao Sun, Chao Ma, Mingli Song

Introduction

Semantic segmentation is a fundamental computer vision task that requires assigning a class label to every pixel in an image. While powerful, training these models is notoriously expensive due to the high cost of pixel-level annotation. Active Learning (AL) offers a solution by automatically selecting the most informative samples from an unlabeled pool for human labeling. However, traditional AL methods often focus on pixel-level uncertainty or dataset diversity, frequently overlooking the inherent semantic difficulty of specific objects, such as slender poles or distant traffic signs.

This paper introduces DEAL (Difficulty-aware Active Learning), a framework designed to explicitly model and prioritize these challenging semantic regions, leading to more efficient and robust segmentation models.

The DEAL Framework

The core of DEAL is a dual-branch architecture consisting of a standard Segmentation Branch (e.g., DeepLabv3+) and a specialized Semantic Difficulty Branch. The difficulty branch is trained to predict where the segmentation branch is likely to fail.

Error Mask Supervision

To train the difficulty branch, the framework generates an "error mask" by comparing the segmentation branch's predictions against the Ground Truth (GT). This binary mask (0 for correct, 1 for incorrect) serves as the supervision signal, teaching the difficulty branch to identify patterns associated with segmentation errors.

Visualizing the components of error mask supervision: Ground Truth (GT), Predicted result, and the derived Error mask.

Probability Attention Module (PAM)

A key innovation in DEAL is the Probability Attention Module (PAM). Instead of treating pixels in isolation, PAM aggregates pixels with similar softmax probabilities. This ensures that difficulty scores are distributed across entire semantic regions rather than being noisy and pixelated. If a model struggles with one part of a complex object, PAM helps it recognize that the entire object area is likely difficult.

A predicted difficulty map highlighting areas where the model is likely to fail, such as around pedestrians and complex boundaries.

Difficulty-aware Acquisition Functions

Once difficulty maps are generated for unlabeled images, DEAL employs two novel strategies for sample selection:

  1. Difficulty-aware uncertainty Score (DS): This metric multiplies standard entropy by the predicted difficulty score, prioritizing pixels that are both uncertain and likely to be part of a hard semantic area.
  2. Difficulty-aware semantic Entropy (DE): This image-level selector quantifies the difficulty map into discrete levels and calculates entropy across them. It favors images with a diverse profile of difficulty levels, providing a more comprehensive learning signal.

Experimental Evaluation

The researchers evaluated DEAL on the Cityscapes and CamVid urban scene benchmarks. Starting with a 10% labeled seed, they iteratively added 5% more data based on AL selection.

Key Results

  • State-of-the-Art Performance: DEAL consistently outperformed standard baselines (Random, Entropy) and complex AL methods (VAAL, Core-set) across all stages.
  • Data Efficiency: On the CamVid dataset, DEAL achieved 95% of the full-dataset performance using only 40% of the labeled data.
  • Improvement on "Hard" Classes: The most significant gains were observed in difficult categories. On Cityscapes, DEAL showed substantial IoU improvements for classes like poles, traffic signs, riders, and motorcycles, which are often neglected by traditional uncertainty-based methods.

Conclusion

DEAL demonstrates that modeling "difficulty" as a semantic property—rather than a random error—significantly enhances active learning for segmentation. By using the model's own mistakes as a teaching tool and smoothing those signals via the Probability Attention Module, DEAL achieves superior performance with a fraction of the annotation effort. Future work may explore region-level selection to further optimize the labeling process.

Summary updated

BibTeX
@inproceedings{xie2020deal,
  author = {Shuai Xie and Zunlei Feng and Ying Chen and Songtao Sun and Chao Ma and Mingli Song},
  title = {{DEAL: Difficulty-Aware Active Learning for Semantic Segmentation}},
  booktitle = {ACCV},
  year = {2020},
  url = {https://openreview.net/forum?id=l8PwBRtrYFs},
}
Figure