Improving Semi-Supervised Semantic Segmentation with Sliced-Wasserstein Feature Alignment and Uniformity

Chen-Yi Lu, Kasra Derakhshandeh, Somali ChaterjiCVPR 2025 · DOI

Introduction

Semantic segmentation is a fundamental computer vision task that requires assigning a class label to every pixel in an image. While highly effective, deep learning models for segmentation are notoriously data-intensive; manually labeling a single image can require over an hour of human effort. Semi-supervised learning (SSL) offers a promising solution by leveraging a small set of labeled data alongside a much larger pool of unlabeled images.

Traditional SSL approaches primarily rely on consistency regularization, which enforces that a model's output remains stable under various input perturbations (e.g., geometric transforms or color jittering). However, these methods typically focus on the final decoder output, often neglecting the intermediate feature representations produced by the encoder backbone. This oversight can lead to "feature collapse," where the model fails to distinguish between different semantic classes in its latent space, resulting in poor performance at complex object boundaries.

The SWSEG Framework

To address these limitations, the authors introduce SWSEG, a novel framework that optimizes the "hidden" feature space of the encoder. SWSEG is built on two core principles of high-quality feature representations borrowed from contrastive learning:

  1. Alignment: Ensuring that different views or perturbations of the same image result in similar feature embeddings.
  2. Uniformity: Ensuring that feature embeddings are distributed evenly across the hypersphere to maximize information preservation and prevent representational collapse.

Efficient Sliced-Wasserstein Distance

The mathematical engine of SWSEG is the Sliced-Wasserstein Distance (SWD). While standard SWD is effective for comparing probability distributions, it is computationally expensive due to its reliance on Monte Carlo sampling. SWSEG overcomes this by utilizing a Gaussian-approximated variant. By projecting features onto a normalized Gaussian distribution, the model achieves a closed-form mathematical solution that is significantly faster and more memory-efficient. This approximation naturally promotes uniformity on the hypersphere. To ensure the validity of this approximation, a decorrelation loss ($\mathcal{L}_{reg}$) is employed to maintain independence between feature dimensions.

Key Results and Evaluation

SWSEG was evaluated on three major benchmarks: PASCAL VOC 2012, Cityscapes, and the highly challenging ADE20K dataset. The model was tested across various labeled data splits (e.g., 1/32, 1/16, 1/8) to assess its robustness in low-supervision scenarios.

mIoU performance of SWSeg compared to other state-of-the-art semi-supervised methods across various data splits.

As shown in the results, SWSEG consistently outperforms both supervised baselines and current state-of-the-art (SOTA) methods:

  • Performance Gains: On PASCAL VOC, SWSEG improved mean Intersection-over-Union (mIoU) by up to 11.8% over the supervised-only baseline.
  • State-of-the-Art Benchmarking: On ADE20K with only 1/32 of the labels, SWSEG achieved 24.4% mIoU, surpassing other leading methods such as DT (22.6%) and ReCo (21.1%).
  • Boundary Precision: Qualitative analysis indicates that SWSEG produces significantly "cleaner" object boundaries compared to methods that only enforce output-level consistency. By refining the encoder's internal representations, the segmentation head can make more accurate pixel-level decisions at the edges of objects.

Conclusion and Future Work

SWSEG demonstrates that optimizing the internal feature space is just as critical as enforcing output consistency in semi-supervised semantic segmentation. By leveraging the efficiency of Gaussian-approximated Sliced-Wasserstein Distance, the model achieves superior performance without the computational overhead of traditional distribution matching techniques.

A current limitation of the approach is its reliance on the Central Limit Theorem for the Gaussian approximation, which assumes weak correlation between features. While the decorrelation loss mitigates this, future research could explore alternative approximations for architectures with inherently high feature correlation. Additionally, extending these principles to more complex tasks like panoptic segmentation remains a promising direction for future work.

Summary updated

BibTeX
@inproceedings{lu2025improving,
  author = {Chen-Yi Lu and Kasra Derakhshandeh and Somali Chaterji},
  title = {{Improving Semi-Supervised Semantic Segmentation with Sliced-Wasserstein Feature Alignment and Uniformity}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01884},
}
Figure