U-MixFormer: UNet-Like Transformer with Mix-Attention for Efficient Semantic Segmentation

Seul-Ki Yeom, Julian von KlitzingWACV 2025

TL;DR

This paper introduces U-MixFormer, a novel semantic segmentation architecture that combines the strengths of U-Net's hierarchical feature processing with the global context modeling of Transformers. By introducing a "mix-attention" mechanism that effectively fuses multi-scale encoder and decoder features, U-MixFormer achieves state-of-the-art performance with improved efficiency compared to existing methods.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for applications like autonomous driving and medical diagnosis. While Convolutional Neural Networks (CNNs) like U-Net excel at capturing local details and hierarchical features, they often struggle with long-range dependencies. Vision Transformers, on the other hand, are adept at modeling global context but can be computationally expensive and may not always capture fine-grained local details as effectively. Existing Transformer-based segmentation methods often rely on computationally intensive feature configurations or lack efficient ways to integrate multi-scale information, leading to a trade-off between accuracy and efficiency. This paper aims to bridge this gap by developing an architecture that leverages the best of both worlds: the hierarchical feature extraction of U-Net and the global context understanding of Transformers, while optimizing for efficiency.

Methods

The core innovation of U-MixFormer lies in its novel decoder architecture and its "mix-attention" mechanism.

  • U-Net-like Transformer Decoder: The architecture adopts a U-Net structure, featuring an encoder-decoder backbone. However, instead of traditional skip connections, U-MixFormer uses lateral connections from the encoder stages as "query" features for its attention modules. This allows the decoder to directly leverage high-level semantic information from the encoder.
  • Mix-Attention Mechanism: This is the key novelty. Unlike standard self-attention (where Q, K, V come from the same source) or cross-attention (where Q comes from one source and K, V from another), mix-attention fuses features from multiple encoder and decoder stages to form the "keys" and "values" (Xkv). The "query" features (Xq) are derived from the lateral connections of the encoder. This fusion allows the attention mechanism to effectively capture context across different levels of granularity, leading to enhanced feature refinement.
  • Hierarchical Feature Fusion: The decoder stages sequentially process features. For each stage, the mix-attention module takes the encoder's lateral connection as queries and a mixed representation of encoder and decoder features as keys and values. This process iteratively refines features, integrating global context with local details.
  • Compatibility with Diverse Encoders: U-MixFormer is designed to be compatible with various popular encoders, including Transformer-based ones like MiT and LVT, as well as CNN-based encoders like MSCAN.

Datasets

The paper evaluates U-MixFormer on two widely used semantic segmentation benchmarks:

  • ADE20K: A large-scale dataset for scene parsing with 150 object categories.
  • Cityscapes: A dataset focused on urban scene understanding, featuring 19 object categories.

The authors also mention evaluating on an augmented version of Cityscapes called Cityscapes-C to assess robustness against image corruption.

Key Insights and Ideas

  • Lateral Connections as Queries: Reimagining U-Net's skip connections as query features for attention is a clever way to inject rich, multi-scale encoder information directly into the decoder's context-aware processing.
  • The Power of "Mix-Attention": By creating a unified representation for keys and values from multiple feature levels, the mix-attention mechanism allows for a more comprehensive understanding of the scene, bridging the gap between local and global context more effectively than traditional attention.
  • U-Net + Transformer Synergy: The paper demonstrates that a hybrid approach, specifically a U-Net-like decoder with Transformer attention, can outperform purely Transformer-based or CNN-based methods by combining their respective strengths.
  • Efficiency Gains: Despite incorporating Transformer mechanisms, U-MixFormer achieves significant improvements in computational efficiency (fewer GFLOPS) while simultaneously boosting accuracy, suggesting a more favorable trade-off than many existing methods.

Key Results

U-MixFormer demonstrates state-of-the-art performance across various configurations and encoder backbones.

  • ADE20K Performance: U-MixFormer-B0 achieves 41.2% mIoU with 6.1M parameters and 6.1 GFLOPS, outperforming SegFormer-B0 and FeedFormer-B0 by 3.8% and 2.0% higher mIoU, respectively, while reducing computation by 27.3% and 21.8%.
  • Cityscapes Performance: On Cityscapes, U-MixFormer achieves 79.0% mIoU with only 101.7 GFLOPS, showing improvements over SegFormer and FeedFormer.
  • Heavy-weight Models: U-MixFormer also outperforms SegFormer with heavier encoders like MiT-B3/4/5, achieving comparable or better mIoU with significantly reduced computation.
  • Robustness: U-MixFormer shows exceptional robustness against various image corruptions on Cityscapes-C, outperforming baseline methods.
  • Qualitative Results: Visualizations show U-MixFormer excelling at segmenting intricate object details and boundaries, particularly in challenging regions.

Limitations and Future Work

  • Inference Latency: The paper acknowledges that U-MixFormer can have a slower inference time compared to other light-weight models. This is attributed to the inherent structure of the U-Net, which requires preserving information through lateral connections.
  • Future Work: To address the inference latency, the authors plan to explore model compression techniques such as pruning and knowledge distillation. They also aim to further evaluate and enhance the model's generalization capability across different data domains to ensure robustness.

Summary updated

BibTeX
@inproceedings{yeom2025u,
  author = {Seul-Ki Yeom and Julian von Klitzing},
  title = {{U-MixFormer: UNet-Like Transformer with Mix-Attention for Efficient Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure