Abstract
Deep Convolutional Neural Networks (DCNNs) for semantic segmentation often suffer from overconfidence, data scarcity, and shortcut learning based on contextual bias. We present Superpixel-mix, a boundary-aware data augmentation strategy designed to improve both accuracy and robustness. By mixing image regions based on object-aware superpixels within a teacher-student consistency framework, our method forces models to prioritize object shape over spurious environmental correlations. Superpixel-mix achieves state-of-the-art results in semi-supervised learning on Cityscapes and Pascal VOC, while significantly enhancing robustness against adverse weather and out-of-context scenarios.
Introduction
While DCNNs excel at standard benchmarks, their real-world reliability is often compromised. Models frequently exhibit high-certainty errors, rely on expensive pixel-level annotations, and fall victim to "shortcut learning"—where they associate objects with specific backgrounds (e.g., assuming a car must be on a road) rather than learning intrinsic object features. This paper addresses these hurdles by introducing a novel augmentation technique that regularizes models against contextual bias and improves performance in data-scarce environments.
Methods
The core innovation is Superpixel-mix, a boundary-aware data augmentation technique. Standard mixing techniques like CutMix swap rectangular patches between images, which often cut through objects and create nonsensical shapes. In contrast, Superpixel-mix uses the Watershed algorithm to group visually similar pixels into "superpixels" that follow natural edges.
To leverage unlabeled data, the authors employ a Teacher-Student framework. A Teacher network generates "pseudo-labels" for unlabeled images. These images and their labels are then mixed using superpixel masks. The Student network is trained to predict the mixed pseudo-labels from the mixed input image. This enforces a consistency constraint, teaching the model that an object's identity should remain stable regardless of its background context.
Results and Key Insights
Evaluation across Cityscapes, Pascal VOC, and StreetHazards demonstrates the effectiveness of Superpixel-mix. In semi-supervised Cityscapes (using only 1/30th of labels), it improves mIoU by +7.27% over the baseline, outperforming methods like Cutmix and Classmix.
Robustness and Reduced Bias
A key contribution of this work is the introduction of OC-Cityscapes (Out-of-Context Cityscapes), where road and pavement textures are replaced with water, sand, or grass. While baseline models show a performance drop of ~22% in these scenarios due to contextual bias, Superpixel-mix remains significantly more resilient.
Furthermore, the method achieves superior performance on Foggy Cityscapes (74.39% mIoU) and Rainy Cityscapes (61.87% mIoU). It also provides competitive out-of-distribution (OOD) detection on the StreetHazards dataset using just a single forward pass, making it more efficient than complex ensemble methods.
Conclusion
Superpixel-mix provides a robust framework for semantic segmentation by emphasizing object shape over environmental context. By respecting object boundaries during augmentation, the method effectively regularizes models against shortcut learning. While it introduces a minor pre-processing step for superpixel generation, the gains in semi-supervised performance and real-world reliability make it a compelling alternative to standard augmentation strategies. Future work may extend this framework to domain adaptation and self-supervised pre-training.
Summary updated