Superpixel Transformers for Efficient Semantic Segmentation

Alex Zihao Zhu, Jieru Mei, Siyuan Qiao, Hang Yan, Yukun Zhu, Liang-Chieh Chen, Henrik Kretzschmar

Abstract

Semantic segmentation is a fundamental task in machine perception, requiring the classification of every pixel in an image. While essential for applications like autonomous driving, the high dimensionality of dense images makes it computationally expensive to capture global context. We present the Superpixel (SP) Transformer, a framework that leverages superpixels—learned over-segmentations of an image—to reduce spatial dimensionality. By applying multi-head self-attention in a significantly smaller superpixel space, our model achieves global context integration with high efficiency. Our method matches state-of-the-art accuracy on Cityscapes and ADE20K while offering superior latency and reduced parameter counts.

Introduction

The goal of semantic segmentation is to assign a semantic label to every pixel. As image resolutions increase, the computational burden grows, particularly for Transformer-based architectures where self-attention complexity scales quadratically with the number of input tokens. To manage this, many existing models rely on local attention windows or dense convolutional decoders, which often fail to capture long-range dependencies or perform redundant calculations on similar neighboring pixels.

In "Superpixel Transformers for Efficient Semantic Segmentation," we propose a shift from pixel-level reasoning to superpixel-level reasoning. By grouping pixels into a sparse set of learned superpixels, we can apply powerful global attention mechanisms at a fraction of the cost.

Methodology

The SP-Transformer architecture is designed to be end-to-end differentiable, allowing the model to learn optimal superpixel groupings for the specific task of segmentation. The pipeline consists of four primary stages:

1. Pixel Feature Extraction

The process begins with a standard backbone (e.g., ResNet-50 or ConvNeXt-L) that extracts high-level features from the input image. These features serve as the foundation for both superpixel formation and final label projection.

2. Superpixel Tokenization

This stage reduces the spatial dimensionality of the feature map. Instead of using traditional clustering algorithms, we employ a series of local dual-path cross-attentions. A set of learned "superpixel queries" interacts with local pixel neighborhoods to aggregate relevant features. This process effectively compresses the dense pixel grid into a sparse set of superpixel tokens—typically reducing the token count by a factor of over 1,000 (e.g., from $1024 \times 2048$ pixels to $32 \times 64$ superpixels).

3. Superpixel Classification

With the data now represented in a compact superpixel space, the model applies standard Multi-Head Self-Attention (MHSA). This allows each superpixel to incorporate global context from the entire scene. A linear layer then predicts the semantic class for each superpixel token.

4. Superpixel Association

The final stage involves projecting the superpixel-level predictions back to the original pixel resolution. This is achieved using the association weights learned during the tokenization phase, ensuring that the final segmentation map respects the original image boundaries.

Experimental Results

We evaluated the SP-Transformer on the Cityscapes and ADE20K benchmarks, focusing on the trade-off between accuracy (mIoU) and efficiency (latency and parameters).

Cityscapes Performance

On the high-resolution Cityscapes dataset, the SP-Transformer demonstrated significant efficiency gains:

  • Accuracy: Using a ResNet-50 backbone, the model achieved 80.4 mIoU.
  • Efficiency: It outperformed the previous state-of-the-art, kMaX-DeepLab, while utilizing only 29M parameters (compared to 56M) and achieving 15.3 FPS on a Tesla V100 (compared to 9.0 FPS).
  • High-End Performance: With a ConvNeXt-L backbone, the model reached 83.1 mIoU while maintaining a competitive frame rate of 3.6 FPS.

ADE20K Performance

The model also showed strong results on the ADE20K dataset, which features 150 semantic classes. While the performance gap to the absolute state-of-the-art was slightly larger than on Cityscapes—likely due to the high class count and inconsistent label boundaries—the SP-Transformer remained highly competitive in terms of accuracy-to-latency ratio.

Discussion

A key insight of this work is that superpixels do not need explicit supervision. The model learns to align superpixel boundaries with physical object boundaries (such as poles, vehicles, and vegetation) simply by optimizing for the final segmentation loss. This "implicit learning" allows the model to preserve fine details that are often lost in other downsampling-based efficient architectures.

However, some limitations remain. The current implementation of local cross-attention relies on standard TensorFlow operations; the authors suggest that custom CUDA kernels could further optimize the tokenization step. Additionally, the performance on ADE20K suggests that very complex scenes may require larger local receptive fields during the tokenization phase to improve grouping accuracy.

Conclusion

The Superpixel Transformer offers a compelling solution to the efficiency challenges of semantic segmentation. By moving reasoning from the dense pixel grid to a learned, sparse superpixel space, the model achieves a state-of-the-art balance of accuracy and speed. This approach paves the way for high-resolution machine perception in resource-constrained environments like autonomous vehicles and mobile robotics.

Summary updated

Abstract

Semantic segmentation, which aims to classify every pixel in an image, is a key task in machine perception, with many applications across robotics and autonomous driving. Due to the high dimensionality of this task, most existing approaches use local operations, such as convolutions, to generate per-pixel features. However, these methods are typically unable to effectively leverage global context information due to the high computational costs of operating on a dense image. In this work, we propose a solution to this issue by leveraging the idea of superpixels, an over-segmentation of the image, and applying them with a modern transformer framework. In particular, our model learns to decompose the pixel space into a spatially low dimensional superpixel space via a series of local cross-attentions. We then apply multi-head self-attention to the superpixels to enrich the superpixel features with global context and then directly produce a class prediction for each superpixel. Finally, we directly project the superpixel class predictions back into the pixel space using the associations between the superpixels and the image pixel features. Reasoning in the superpixel space allows our method to be substantially more computationally efficient compared to convolution-based decoder methods. Yet, our method achieves state-of-the-art performance in semantic segmentation due to the rich superpixel features generated by the global self-attention mechanism. Our experiments on Cityscapes and ADE20K demonstrate that our method matches the state of the art in terms of accuracy, while outperforming in terms of model parameters and latency.
BibTeX
@inproceedings{zhu2023superpixel,
  author = {Alex Zihao Zhu and Jieru Mei and Siyuan Qiao and Hang Yan and Yukun Zhu and Liang-Chieh Chen and Henrik Kretzschmar},
  title = {{Superpixel Transformers for Efficient Semantic Segmentation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=vFqVgOVE92},
}
Figure