Training Vision Transformers for Semi-Supervised Semantic Segmentation

Xinting Hu, Li Jiang, Bernt Schiele

Introduction

Semantic segmentation—the task of assigning a class label to every pixel in an image—is a cornerstone of computer vision, essential for applications ranging from medical imaging to autonomous driving. However, the performance of these models is heavily dependent on large-scale, pixel-perfect annotations, which are notoriously expensive and time-consuming to produce. Semi-supervised semantic segmentation ($S^4$) addresses this bottleneck by leveraging a small set of labeled data alongside a much larger pool of unlabeled images.

While Vision Transformers (ViTs) have largely superseded Convolutional Neural Networks (ConvNets) in fully supervised tasks, their adoption in the semi-supervised domain has lagged. Simple "plug-and-play" replacements of ConvNet backbones with ViTs often yield disappointing results. This research introduces $S^4$Former, a framework specifically designed to unlock the potential of ViTs for semi-supervised learning through "Transformer-native" regularization.

The $S^4$Former Framework

$S^4$Former adopts the established Teacher-Student paradigm, where a "Teacher" model generates pseudo-labels for unlabeled data to guide a "Student" model. The core innovation lies in how the Student is regularized to handle the inherent noise in these pseudo-labels while exploiting the unique architectural properties of Transformers.

Overview of the S4Former framework, illustrating the interaction between the Teacher and Student models and the integration of PatchShuffle, PASA, and NCR loss.

The framework introduces three primary components:

  1. PatchShuffle: Unlike traditional spatial augmentations (like rotation or scaling), PatchShuffle operates on the grid of patches that form the input to a ViT. By shuffling these blocks, the model is forced to rely on global contextual relationships rather than local spatial continuity to identify objects, significantly enhancing its robustness.
  2. Patch-Adaptive Self-Attention (PASA): This mechanism uses the Teacher's confidence scores to modulate the Student's attention. By identifying low-confidence (ambiguous) regions, PASA directs the Student's focus toward the most challenging parts of the image, effectively "teaching" the model where it needs to learn most.
  3. Negative Class Ranking (NCR) Loss: While standard cross-entropy loss focuses on the target class, NCR loss regularizes the entire probability distribution. It ensures the Student's ranking of "incorrect" classes matches the Teacher's, providing a richer supervision signal that prevents overconfidence in erroneous predictions.

Key Results and Benchmarks

The effectiveness of $S^4$Former was validated across several major benchmarks, including Pascal VOC 2012, COCO, and Cityscapes. The results demonstrate a significant leap over previous state-of-the-art methods, particularly in data-scarce regimes.

Comparison of S4Former against previous state-of-the-art methods. S4Former (using SETR and DeiT-Base) achieves a new high-water mark in mIoU performance.

On the Pascal VOC 2012 dataset with only 92 labeled images, $S^4$Former achieved an 80.1% mIoU, representing a 4.9% improvement over the previous best-performing model. Similar gains were observed on the more complex COCO dataset. Notably, these performance improvements are achieved without increasing the computational cost during inference, as the novel components are only active during the training phase.

Conclusion

$S^4$Former demonstrates that the key to successful semi-supervised learning with Vision Transformers lies in architecture-specific regularization. By respecting the patch-based nature and global attention mechanisms of ViTs, $S^4$Former sets a new standard for the field, paving the way for more efficient and powerful segmentation models that require far less manual annotation. Future work may explore the integration of these techniques into even larger "foundation" models to further push the boundaries of what is possible with limited labeled data.

Summary updated

Abstract

We present S 4 Former, a novel approach to training Vision Transformers for Semi-Supervised Semantic Segmentation (S4). At its core, S 4 Former employs a Vision Transformer within a classic teacher-student framework, and then leverages three novel technical ingredients: PatchShuffle as a parameter-free perturbation technique, Patch-Adaptive Self-Attention (PASA) as a fine-grainedfeature modulation method, and the innovative Negative Class Ranking (NCR) regularization loss. Based on these regu-larization modules aligned with Transformer-specific char-acteristics across the image input, feature, and output di-mensions, S4Former exploits the Transformer's ability to capture and differentiate consistent global contextual information in unlabeled images. Overall, S4 Former not only defines a new state of the art in S4 but also maintains a streamlined and scalable architecture. Being readily compatible with existing frameworks, S4 Former achieves strong improvements (up to 4.9%) on benchmarks like Pascal VOC 2012, COCO, and Cityscapes, with varying numbers of labeled data. The code is at https://github.com/JoyHuYY1412/S4Former.
BibTeX
@inproceedings{hu2024training,
  author = {Xinting Hu and Li Jiang and Bernt Schiele},
  title = {{Training Vision Transformers for Semi-Supervised Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00384},
}
Figure