Superpixel Transformers for Efficient Semantic Segmentation

Alex Zihao Zhu, Jieru Mei, Siyuan Qiao, Hang Yan, Yukun Zhu, Liang-Chieh Chen, Henrik Kretzschmar

Semantic segmentation—the task of assigning a category to every single pixel in an image—is the "bread and butter" of computer vision for autonomous driving and robotics. However, as image resolutions climb, the computational cost of processing millions of pixels individually becomes a massive bottleneck.

In their paper, "Superpixel Transformers for Efficient Semantic Segmentation," researchers from Waymo, Google, and partner institutions propose a clever way to slash these costs: stop treating every pixel as an individual and start reasoning about "superpixels."

TL;DR

The Superpixel (SP) Transformer replaces expensive dense pixel-level decoders with a transformer that operates on a much smaller set of learned superpixels. By reducing the spatial dimensionality by over 1,000x (e.g., from $1024 \times 2048$ pixels to $32 \times 64$ superpixels), the model achieves state-of-the-art accuracy on Cityscapes while being significantly faster and lighter than previous methods.

What Problem Does This Paper Address?

Modern semantic segmentation models usually follow an encoder-decoder structure. The encoder extracts features, and the decoder projects them back to the original image resolution to make pixel-wise predictions.

The "Transformer revolution" has brought global context to this field, but transformers are notoriously memory-hungry—their computational cost grows quadratically with the number of inputs. To handle high-resolution images, most models either use local attention windows (which lose global context) or expensive dense decoders that produce redundant computations for neighboring pixels that look nearly identical.

The authors revisit a classical computer vision concept: superpixels. Superpixels are irregular groups of pixels that share similar colors or textures. If you can group pixels into superpixels first, you can perform your heavy reasoning on a few thousand superpixels instead of millions of pixels.

Methods

The SP-Transformer architecture consists of four main stages:

  1. Pixel Feature Extraction: A standard backbone (like ResNet-50 or ConvNeXt-L) extracts "hypercolumn" features from the image.
  2. Superpixel Tokenization: This is the "secret sauce." Instead of using a fixed clustering algorithm, the model uses a series of local dual-path cross-attentions. A set of learned "superpixel queries" looks at local pixel neighborhoods to "grab" relevant features. This converts the dense pixel grid into a sparse set of superpixel tokens.
  3. Superpixel Classification: Now that the data is in "superpixel space" (which is roughly $32^2$ times smaller than the pixel space), the model applies standard Multi-Head Self-Attention. This allows every superpixel to talk to every other superpixel, gaining a global understanding of the scene. A simple linear layer then predicts the class for each superpixel.
  4. Superpixel Association: Finally, the model projects these superpixel-level predictions back to the pixel level using the learned associations from step 2, resulting in a high-resolution segmentation map.

What’s new? Unlike older superpixel methods (like SLIC), this approach is entirely differentiable, meaning the model learns how to form the best superpixels for segmentation directly from the data, without needing explicit "superpixel labels."

Datasets

The authors evaluated their model on two heavy-hitting benchmarks:

  • Cityscapes: A high-resolution ($1024 \times 2048$) dataset of urban street scenes.
  • ADE20K: A general scene parsing dataset with 150 semantic classes and diverse environments.

Key Insights and Ideas

  • Implicit Learning: The model isn't told what a "good" superpixel looks like. Yet, visualizations show that the learned superpixels naturally align with object boundaries (like poles, cars, and trees) simply by trying to minimize the segmentation loss.
  • Global Context on the Cheap: Because the transformer operates on a low-dimensional superpixel space, it can afford to use global self-attention even on high-resolution images. This is often too expensive for pixel-based transformers.
  • Efficiency through Sparsity: By reasoning at the superpixel level, the model discards the redundant computation usually spent on "flat" areas of an image (like a large patch of road or sky) where all pixels belong to the same class.

Key Results

The SP-Transformer punches well above its weight class:

  • Cityscapes: Using a ResNet-50 backbone, it achieved 80.4 mIoU, outperforming the previous state-of-the-art (kMaX-DeepLab) while being half the size (29M vs 56M parameters) and 70% faster (15.3 FPS vs 9.0 FPS).
  • Latency: On a Tesla V100 GPU, the SP-Transformer with a ConvNeXt-L backbone achieved competitive accuracy (83.1 mIoU) while maintaining a higher frame rate (3.6 FPS) than comparable heavy-duty models.
  • Qualitative Quality: Despite operating in a highly compressed superpixel space, the model is remarkably good at recovering thin structures like poles and signs, which are often lost in other efficient architectures.

Limitations and Future Work

  • The ADE20K Gap: While the model performed well on ADE20K, the authors noted a slight performance gap compared to the very best models. They attribute this to the high number of classes (150) and inconsistent label boundaries in the dataset, which can confuse the local superpixel grouping.
  • Implementation Bottlenecks: The current implementation uses a "naive" TensorFlow approach for local cross-attention. The authors believe a custom CUDA kernel could make the "Tokenization" step even faster, potentially unlocking even higher frame rates.
  • Receptive Field: Each superpixel only "sees" a local neighborhood during the tokenization phase. Increasing this neighborhood size could improve results but would increase computational cost.

Summary updated

Abstract

Semantic segmentation, which aims to classify every pixel in an image, is a key task in machine perception, with many applications across robotics and autonomous driving. Due to the high dimensionality of this task, most existing approaches use local operations, such as convolutions, to generate per-pixel features. However, these methods are typically unable to effectively leverage global context information due to the high computational costs of operating on a dense image. In this work, we propose a solution to this issue by leveraging the idea of superpixels, an over-segmentation of the image, and applying them with a modern transformer framework. In particular, our model learns to decompose the pixel space into a spatially low dimensional superpixel space via a series of local cross-attentions. We then apply multi-head self-attention to the superpixels to enrich the superpixel features with global context and then directly produce a class prediction for each superpixel. Finally, we directly project the superpixel class predictions back into the pixel space using the associations between the superpixels and the image pixel features. Reasoning in the superpixel space allows our method to be substantially more computationally efficient compared to convolution-based decoder methods. Yet, our method achieves state-of-the-art performance in semantic segmentation due to the rich superpixel features generated by the global self-attention mechanism. Our experiments on Cityscapes and ADE20K demonstrate that our method matches the state of the art in terms of accuracy, while outperforming in terms of model parameters and latency.
BibTeX
@inproceedings{zhu2023superpixel,
  author = {Alex Zihao Zhu and Jieru Mei and Siyuan Qiao and Hang Yan and Yukun Zhu and Liang-Chieh Chen and Henrik Kretzschmar},
  title = {{Superpixel Transformers for Efficient Semantic Segmentation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=vFqVgOVE92},
}

Loading related content...

Figure