Abstract
Since the fully convolutional network has achieved great success in semantic segmentation, lots of works have been proposed focusing on extracting discriminative pixel feature representations. However, we observe that existing methods still suffer from two typical challenges, i.e. (i) large intra-class feature variation in different scenes, (ii) small inter-class feature distinction in the same scene. In this paper, we first rethink semantic segmentation from a perspective of similarity between pixels and class centers. Each weight vector of the segmentation head represents its corresponding semantic class in the whole dataset, which can be regarded as the embedding of the class center. Thus, the pixel-wise classification amounts to computing similarity in the final feature space between pixels and the class centers. Under this novel view, we propose a Class Center Similarity layer (CCS layer) to address the above-mentioned challenges by generating adaptive class centers conditioned on different scenes and supervising the similarities between class centers. It utilizes an Adaptive Class Center Module (ACCM) to generate class centers conditioned on each scene, which adapt the large intra-class variation between different scenes. Specially designed loss functions are introduced to control both inter-class and intra-class distances based on predicted center-to-center and pixel-to-center similarity, respectively. Finally, the CCS layer outputs the processed pixel-to-center similarity as the segmentation prediction. Extensive experiments demonstrate that our model performs favourably against the state-of-the-art CNN-based methods.
Introduction
Traditional semantic segmentation frameworks, primarily based on Fully Convolutional Networks (FCNs), typically employ a $1 \times 1$ convolutional layer as the final classification head. In this setup, the weights of the convolution act as "global class centers"—fixed templates learned across the entire dataset. However, this approach faces two significant hurdles:
- Large Intra-class Variation: A single global representation for a class (e.g., "tree") often fails to capture the vast visual differences across different environments, such as a snowy forest versus a tropical park.
- Small Inter-class Distinction: Within a specific scene, distinct classes like "tree" and "plant" may appear nearly identical in texture and color, leading to misclassification when using rigid, dataset-wide averages.
This paper proposes rethinking segmentation as a similarity matching problem. By replacing fixed classification heads with dynamic, scene-adaptive centers, models can better distinguish between similar classes while remaining robust to environmental variations.
Methodology: The Class Center Similarity (CCS) Layer
The core contribution is the Class Center Similarity (CCS) layer, a plug-and-play module designed to replace standard segmentation heads. It shifts the paradigm from static classification to dynamic similarity matching through three integrated components.
Adaptive Class Center Module (ACCM)
Instead of relying on fixed weights, the ACCM generates class centers conditioned on the input image. It uses a learned spatial weight map ("Mask") to aggregate features into coarse centers, which are then refined into adaptive vectors. This allows the model to "specialize" its class representations for each unique scene, effectively addressing the intra-class variation problem.
Similarity Calculation and Supervision
The Similarity Calculation Module (SCM) computes the inner product between pixel features and these adaptive centers. To ensure these centers remain discriminative, the authors introduce the Class Distance (CD) Loss. This loss function includes an intra-class component to pull pixels toward their respective adaptive centers and an inter-class component to push different class centers apart, maximizing their linear independence and addressing the inter-class distinction challenge.
Experimental Evaluation
The proposed CCSNet was evaluated on the ADE20K and Pascal Context benchmarks, demonstrating superior performance over existing CNN-based architectures.
- ADE20K: CCSNet achieved 47.76% mIoU with a ResNet-101 backbone, outperforming DeepLabV3+ (46.35%) and OCRNet (45.28%).
- Pascal Context: The model reached 54.9% mIoU, surpassing previous leading models.
Beyond numerical metrics, qualitative analysis via t-SNE visualizations reveals that CCSNet produces significantly tighter and more distinct feature clusters compared to baseline models. This confirms that the adaptive centers effectively resolve ambiguities in complex scenes.
Conclusion
By reframing semantic segmentation as a pixel-to-center similarity task, this work addresses the inherent limitations of fixed classification heads. The CCS layer provides a computationally efficient way to generate scene-adaptive representations, leading to more discriminative features. Future work may involve exploring the synergy between the CCS layer and modern Vision Transformer (ViT) backbones to further push the boundaries of scene parsing.
Summary updated