Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning

Jianjian Yin, Tao Chen, Gensheng Pei, Yazhou Yao, Liqiang Nie, Xian-Sheng HuaIEEE Trans. Multimedia 2025 · DOI · OpenReview

TL;DR

This paper introduces a novel Multi-Constraint Consistency Learning (MCCL) framework for semi-supervised semantic segmentation. MCCL enhances both the encoder and decoder stages by focusing on feature consistency and feature perturbation, leading to state-of-the-art performance on challenging benchmarks.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for many computer vision applications. While fully supervised methods achieve high accuracy, they require extensive pixel-level annotations, which are costly and time-consuming to obtain. Semi-supervised methods aim to leverage large amounts of unlabeled data alongside limited labeled data. Existing semi-supervised semantic segmentation methods primarily focus on prediction consistency between augmented versions of an image. However, the authors argue that this approach underutilizes the rich supervisory information available in the data, particularly by neglecting feature consistency and feature perturbation for more robust learning. They aim to bridge this gap by developing a method that explicitly enforces consistency at both the feature level and through controlled perturbations.

Methods

The proposed Multi-Constraint Consistency Learning (MCCL) approach enhances the segmentation pipeline in a staged manner, targeting both the encoder and decoder.

  • Feature Knowledge Alignment (FKA): This strategy aims to improve the encoder's ability to learn consistent features from differently augmented images. It operates on two principles:
  • Point-to-point alignment: Encourages features at corresponding spatial locations in strongly and weakly augmented views to be similar.
  • Prototype-based intra-class feature compactness: Focuses on aligning features that are close to class prototypes and identifying/compacting outlier features, making the network more robust to variations.
  • Self-Adaptive Intervention (SAI) Module: This module targets the decoder and introduces feature-perturbation based prediction consistency. It aims to increase the discrepancy between features fed to the decoder, forcing it to learn more robustly.
  • Self-adaptive feature masking: Intelligently masks highly activated regions in the strongly augmented feature map based on inter-view feature similarity, preventing overfitting.
  • Self-adaptive noise injection: Injects instance-specific noise into features to diversify them and expand the decoder's learning range.

These interventions are guided by predictions from the weakly augmented view, ensuring that Feature-perturbation based Prediction (FP) consistency is achieved while implicitly maintaining Image-augmentation based Prediction (IP) consistency.

Datasets

The authors evaluate their method on the following datasets:

  • Pascal VOC2012: A standard semantic segmentation dataset with 20 foreground classes and 1 background class. They use both the original dataset and a "blended" version which includes more coarsely labeled images.
  • Cityscapes: A dataset focused on urban scenes, containing 19 classes.

Key Insights and Ideas

  • Feature consistency is crucial: The paper highlights that simply enforcing prediction consistency is insufficient. Aligning features from strongly and weakly augmented views (FKA) significantly improves performance.
  • Targeted feature perturbation for the decoder: Instead of just relying on image augmentations, perturbing features fed to the decoder (SAI) makes it more robust and capable of handling variations.
  • Staged enhancement: Applying different consistency constraints to the encoder (FKA) and decoder (SAI) separately allows for more effective and targeted improvements.
  • Instance-specific interventions: The SAI module's ability to adapt masking and noise injection based on individual instance features is key to its effectiveness.
  • Balancing similarity and discrepancy: The FKA strategy leverages feature similarity, while the SAI module introduces controlled feature discrepancy, creating a powerful complementary effect.

Key Results

The proposed MCCL method achieves state-of-the-art performance on both the Pascal VOC2012 and Cityscapes datasets across various labeled data ratios.

  • On the original Pascal VOC2012 dataset, MCCL with a ResNet-50 backbone achieves 79.62% mIoU in the "Full" setting (1464 labeled images), surpassing UniMatch by 0.92%. With a Transformer backbone, it reaches 81.96% mIoU.
  • On the blended Pascal VOC2012 dataset, MCCL with ResNet-50 outperforms ESL by up to 2.65% mIoU and CCVC by up to 1.56% mIoU.
  • On the Cityscapes dataset, MCCL with a Transformer backbone achieves 80.87% mIoU in the 1/2 setting, outperforming SemiCVT by 1.32% mIoU.

Ablation studies confirm the effectiveness of each component: FKA improves performance by 0.73% mIoU, and SAI adds another 0.59% mIoU over the baseline.

Limitations and Future Work

  • Failure Cases: The paper acknowledges limitations in segmenting regions with similar appearances (e.g., background vs. person) and accurately segmenting elongated objects (poles), as shown in Figure 9.
  • Hyperparameter Sensitivity: While the paper explores hyperparameter tuning (scaling factor $\lambda$, loss weights $\alpha, \omega, \beta$), optimal values might still require careful selection.
  • Future Work: The authors suggest that their staged enhancement approach, combining feature consistency and perturbation, provides a strong foundation for future research in semi-supervised semantic segmentation. The exploration of different probability distribution measurement ways for loss functions (KL divergence, CE, MSE) indicates potential for further optimization.

Summary updated

Abstract

Consistency regularization has prevailed in semi-supervised semantic segmentation and achieved promising performance. However, existing methods typically concentrate on enhancing the Image-augmentation based Prediction consistency and optimizing the segmentation network as a whole, resulting in insufficient utilization of potential supervisory information. In this paper, we propose a Multi-Constraint Consistency Learning (MCCL) approach to facilitate the staged enhancement of the encoder and decoder. Specifically, we first design a feature knowledge alignment (FKA) strategy to promote the feature consistency learning of the encoder from image-augmentation. Our FKA encourages the encoder to derive consistent features for strongly and weakly augmented views from the perspectives of point-to-point alignment and prototype-based intra-class compactness. Moreover, we propose a self-adaptive intervention (SAI) module to increase the discrepancy of aligned intermediate feature representations, promoting Feature-perturbation based Prediction consistency learning. Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder. Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance. The source code and models are made available at https://github.com/NUST-Machine-Intelligence-Laboratory/MCCL.
BibTeX
@article{yin2025semi,
  author = {Jianjian Yin and Tao Chen and Gensheng Pei and Yazhou Yao and Liqiang Nie and Xian-Sheng Hua},
  title = {{Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3586111},
}
Figure