Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition

Jiebin Yan, Yu Zhong, Yuming Fang, Zhangyang Wang, Kede Ma

Introduction

Current progress in semantic segmentation is primarily measured by performance on static, human-labeled benchmarks such as PASCAL VOC or Cityscapes. While these datasets have driven significant innovation, they represent "closed worlds." Models often achieve high scores by exploiting dataset-specific biases or "shortcuts" rather than developing a robust, generalizable understanding of the visual world. Testing on the "open world" is hindered by the prohibitive cost of pixel-level human labeling. This paper introduces a framework to efficiently identify model "blind spots" by leveraging a Maximum Discrepancy (MAD) competition between competing algorithms.

Methodology: The MAD Competition

The authors propose a "defender-attacker" game to evaluate model generalizability under a strict human labeling budget. Instead of random sampling, the framework automatically searches a massive unlabeled image pool to find instances where two models disagree most significantly.

The MAD competition pipeline: (a) 100k+ unlabeled image pool, (b) selection of high-discrepancy images with category/scale constraints, (c) comparison of model predictions against ground truth.

The process involves:

  1. The Search: From a pool of over 100,000 unlabeled web images, the system selects a small subset (the MAD set) that maximizes the prediction discrepancy between model pairs.
  2. Diversity Constraints: To ensure a representative evaluation, the search enforces category and scale constraints, preventing the selection of redundant error types.
  3. Human-in-the-loop: Only the high-discrepancy subset is sent for human labeling, drastically reducing annotation costs.
  4. Ranking Metrics: Models are evaluated on Aggressiveness (ability to find competitor failures) and Resistance (accuracy on images selected to expose their own weaknesses).

Experimental Setup

The study utilized an unlabeled pool of 100,000+ images crawled from the web using PASCAL VOC keywords. Ten state-of-the-art models, including DeepLabv3+, EMANet, and PSPNet, were trained on PASCAL VOC and then subjected to the MAD competition. The final MAD evaluation set comprised 833 highly informative images. Validation was also performed on the WildDash dataset for autonomous driving scenarios.

Analysis of Model Robustness

The MAD competition revealed that standard benchmarks can be deceptive. Models maintaining >80% mIoU on PASCAL VOC saw their performance crash on the MAD set; 84% of the selected images resulted in an mIoU below 0.6.

Example of a segmentation failure on the MAD set: A complex boat scene where the model's prediction (right) significantly deviates from the ground truth (left).

Key insights include:

  • The Attention Trap: Models utilizing self-attention modules (e.g., EMANet, PSANet) often dropped in rank during the MAD competition. This suggests that while attention helps on standard benchmarks, it may rely on delicate features that fail to generalize to the open world.
  • Multi-tasking Benefits: BlitzNet, which integrates object detection and segmentation, showed superior robustness. The detection task likely acts as a regularizer, encouraging the learning of more stable, localized features.
  • Failure Transferability: Images that cause disagreement between two models often expose general weaknesses that affect other models as well, acting as "naturally occurring adversarial examples."

Results and Discussion

While DeepLabv3+ remained the top performer, the MAD rankings significantly reordered the "middle of the pack." Robust models like RefineNet and BlitzNet climbed 3 to 5 spots compared to their standard benchmark rankings. Conversely, attention-heavy models showed lower "Resistance" to attacks. In autonomous driving tests, HANet's superior performance highlighted that data diversity remains a critical factor for generalization.

Conclusion and Future Work

The MAD competition framework provides a scalable way to stress-test computer vision models beyond the limits of static benchmarks. Future work includes using MAD-selected images for iterative fine-tuning to harden models against identified failures and exploring more cost-effective annotation types, such as bounding boxes, to further expand the evaluation scale.

Summary updated

BibTeX
@article{yan2021exposing,
  author = {Jiebin Yan and Yu Zhong and Yuming Fang and Zhangyang Wang and Kede Ma},
  title = {{Exposing Semantic Segmentation Failures via Maximum Discrepancy Competition}},
  journal = {Int. J. Comput. Vis.},
  year = {2021},
  doi = {10.1007/s11263-021-01450-2},
}
Figure