Abstract
Semantic segmentation is a fundamental task in machine perception, requiring the classification of every pixel in an image. While essential for applications like autonomous driving, the high dimensionality of dense images makes it computationally expensive to capture global context. We present the Superpixel (SP) Transformer, a framework that leverages superpixels—learned over-segmentations of an image—to reduce spatial dimensionality. By applying multi-head self-attention in a significantly smaller superpixel space, our model achieves global context integration with high efficiency. Our method matches state-of-the-art accuracy on Cityscapes and ADE20K while offering superior latency and reduced parameter counts.
Introduction
The goal of semantic segmentation is to assign a semantic label to every pixel. As image resolutions increase, the computational burden grows, particularly for Transformer-based architectures where self-attention complexity scales quadratically with the number of input tokens. To manage this, many existing models rely on local attention windows or dense convolutional decoders, which often fail to capture long-range dependencies or perform redundant calculations on similar neighboring pixels.
In "Superpixel Transformers for Efficient Semantic Segmentation," we propose a shift from pixel-level reasoning to superpixel-level reasoning. By grouping pixels into a sparse set of learned superpixels, we can apply powerful global attention mechanisms at a fraction of the cost.
Methodology
The SP-Transformer architecture is designed to be end-to-end differentiable, allowing the model to learn optimal superpixel groupings for the specific task of segmentation. The pipeline consists of four primary stages:
1. Pixel Feature Extraction
The process begins with a standard backbone (e.g., ResNet-50 or ConvNeXt-L) that extracts high-level features from the input image. These features serve as the foundation for both superpixel formation and final label projection.
2. Superpixel Tokenization
This stage reduces the spatial dimensionality of the feature map. Instead of using traditional clustering algorithms, we employ a series of local dual-path cross-attentions. A set of learned "superpixel queries" interacts with local pixel neighborhoods to aggregate relevant features. This process effectively compresses the dense pixel grid into a sparse set of superpixel tokens—typically reducing the token count by a factor of over 1,000 (e.g., from $1024 \times 2048$ pixels to $32 \times 64$ superpixels).
3. Superpixel Classification
With the data now represented in a compact superpixel space, the model applies standard Multi-Head Self-Attention (MHSA). This allows each superpixel to incorporate global context from the entire scene. A linear layer then predicts the semantic class for each superpixel token.
4. Superpixel Association
The final stage involves projecting the superpixel-level predictions back to the original pixel resolution. This is achieved using the association weights learned during the tokenization phase, ensuring that the final segmentation map respects the original image boundaries.
Experimental Results
We evaluated the SP-Transformer on the Cityscapes and ADE20K benchmarks, focusing on the trade-off between accuracy (mIoU) and efficiency (latency and parameters).
Cityscapes Performance
On the high-resolution Cityscapes dataset, the SP-Transformer demonstrated significant efficiency gains:
- Accuracy: Using a ResNet-50 backbone, the model achieved 80.4 mIoU.
- Efficiency: It outperformed the previous state-of-the-art, kMaX-DeepLab, while utilizing only 29M parameters (compared to 56M) and achieving 15.3 FPS on a Tesla V100 (compared to 9.0 FPS).
- High-End Performance: With a ConvNeXt-L backbone, the model reached 83.1 mIoU while maintaining a competitive frame rate of 3.6 FPS.
ADE20K Performance
The model also showed strong results on the ADE20K dataset, which features 150 semantic classes. While the performance gap to the absolute state-of-the-art was slightly larger than on Cityscapes—likely due to the high class count and inconsistent label boundaries—the SP-Transformer remained highly competitive in terms of accuracy-to-latency ratio.
Discussion
A key insight of this work is that superpixels do not need explicit supervision. The model learns to align superpixel boundaries with physical object boundaries (such as poles, vehicles, and vegetation) simply by optimizing for the final segmentation loss. This "implicit learning" allows the model to preserve fine details that are often lost in other downsampling-based efficient architectures.
However, some limitations remain. The current implementation of local cross-attention relies on standard TensorFlow operations; the authors suggest that custom CUDA kernels could further optimize the tokenization step. Additionally, the performance on ADE20K suggests that very complex scenes may require larger local receptive fields during the tokenization phase to improve grouping accuracy.
Conclusion
The Superpixel Transformer offers a compelling solution to the efficiency challenges of semantic segmentation. By moving reasoning from the dense pixel grid to a learned, sparse superpixel space, the model achieves a state-of-the-art balance of accuracy and speed. This approach paves the way for high-resolution machine perception in resource-constrained environments like autonomous vehicles and mobile robotics.
Summary updated