LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter

Tobias Christian Nauen, Anosh Billimoria, Federico Raue, Stanislav Frolov, Brian B. Moser, Andreas DengelarXiv 2026 · arXiv

TL;DR

Evaluating vision backbones for image segmentation has long been an unfair game, as each model is typically paired with a different decoder and training recipe. This paper introduces LUMA (Lightweight Universal Mask Adapter), a minimal, backbone-agnostic head that acts as a fair "measuring stick" to benchmark 20 transformer architectures and 11 pretraining schemes. The surprising verdict? Plain ViT remains the efficiency king, and the pretraining objective matters far more than clever architectural tweaks.

What Problem Does This Paper Address?

In computer vision, designing "efficient" token mixers (like Swin, Linformer, or specialized convolutional hybrids) is a popular research direction aimed at avoiding the quadratic cost of standard attention at high resolutions. However, when these backbones are evaluated on downstream tasks like semantic segmentation, they are bundled with heavy, task-specific decoders (e.g., Mask2Former) and custom training recipes. This confounds the results: is a high mIoU score due to a superior backbone, a massive decoder, or a lucky pretraining setup?

Previous attempts to strip away the decoder clutter, such as EoMT, only worked on plain, isotropic ViTs because they embedded segmentation queries directly into the backbone's global self-attention mechanism. This paper addresses the need for a universal, lightweight, and non-intrusive segmentation head that can attach to any backbone architecture—isotropic, hierarchical, convolutional, or mixture-of-experts—allowing researchers to isolate and fairly evaluate the backbone's true capabilities.

Methods

The authors introduce the Lightweight Universal Mask Adapter (LUMA). Instead of forcing segmentation queries into the backbone's internal token stream, LUMA keeps them in a separate side-stream.

Figure 1. LUMA architecture. The backbone (gray) runs unmodi- fied on the N patch tokens. A side-stream of K queries reads from each tapped block’s features via cross-attention, then passes queries through the backbone’s own MLP block.

Key methodological highlights include:

  • Black-Box Interaction: LUMA reads features from the backbone's blocks using a single, cheap cross-attention layer per tapped block. The backbone runs exactly as originally designed.
  • Weight and Schedule Reuse: After cross-attention, the queries are processed using the backbone's own existing Feed-Forward Network (MLP) layers. This means queries automatically inherit the backbone's width and dimension schedule without requiring redundant, architecture-specific projection layers.
  • Unmasked Simplicity: Unlike prior work that relies on complex masked-attention and annealing schedules during training, LUMA achieves state-of-the-art accuracy using plain, unmasked cross-attention, significantly simplifying the training pipeline.

Datasets

The unified benchmark evaluates the models on two standard segmentation datasets:

  • ADE20K
  • Cityscapes

Using LUMA as the fixed head, the authors benchmarked 20 distinct backbone architectures, 11 pretraining schemes, and multiple image resolutions under a single modern training recipe.

Key Insights and Ideas

  • ImageNet Accuracy is a Poor Proxy: A backbone's top-1 accuracy on ImageNet-1k is an unreliable predictor of its final segmentation performance. Many specialized or convolutional hybrid models fall well below the expected trend line when moving to dense prediction tasks.
  • The "Efficient" Token Mixer Illusion: Many architectures designed specifically to be "efficient" at high resolutions fail to deliver actual throughput benefits. When evaluated fairly, plain ViT (especially when boosted by FlashAttention) consistently holds the Pareto-front for speed and accuracy. Specialized mixers often only offer advantages in memory consumption, not speed.
  • Pretraining Trumps Architecture: The pretraining objective is the strongest lever for segmentation quality. Dense self-supervised objectives (like DINOv2/v3 or masked image modeling via EVA-02) transfer robustly to dense tasks, whereas standard supervised pretraining (like DeiT III) performs poorly, sitting up to 8 percentage points lower in mIoU.
  • The 80/20 Rule for Scaling: When scaling up a compute budget, the optimal path is highly asymmetric. The authors' compute-optimal scaling analysis shows that roughly 80% of additional compute should be allocated to increasing model parameter size, while only 20% should go toward increasing input resolution.

Key Results

  • State-of-the-Art Efficiency: Paired with a canonical ViT-L/DINOv2 backbone, LUMA achieves 84.4 mIoU on Cityscapes and 58.7 mIoU on ADE20K. This matches or exceeds heavy decoder baselines while using 1.3× fewer GFLOPs and running over 7× faster than a ViT-Adapter + Mask2Former setup.
  • High-Resolution Collapse: Several sequence-length-bound mixers fail to generalize to the high resolutions required for segmentation. For instance, Linformer's performance collapses from 38.4 mIoU down to 3.8 mIoU as resolution scales up, a critical flaw that classification benchmarks completely hide.
Figure 3. Throughput vs. ADE20K mIoU across backbone taxonomy. Panels group families from [41]; markers indicate image resolution. Plain ViT (with FlashAttention) holds the Pareto front; among efficient backbones mostly sparse attention (Swin-S, WaveViT) is close. Several mixers degrade as resolution rises.

Limitations and Future Work

  • Pretraining Availability Confound: While the paper concludes that dense pretraining is superior, dense-objective pretrained weights are currently only widely available for plain ViT architectures. Consequently, the broad 20-backbone architectural comparison had to rely on a shared supervised pretraining setup.
  • Resolution Saturation: The benefits of resolution scaling are strictly bounded by the native resolution of the dataset's images, meaning resolution scaling eventually hits a hard ceiling.
  • Dataset Quirks: Scaling behaviors can be dataset-dependent; for example, the massive ViT-g model proved compute-optimal on the challenging ADE20K dataset but was suboptimal on Cityscapes compared to smaller models running at higher resolutions.

Summary updated

Abstract

Comparing transformer backbones for image segmentation is confounded: each is paired with a different decoder, recipe, and pretraining, so reported differences rarely reflect the backbone itself. We introduce the Lightweight Universal Mask Adapter (LUMA), a lightweight, backbone-agnostic mask-transformer head that treats any backbone as a black-box feature extractor, letting a set of queries read from its features through cheap cross-attention. LUMA matches the accuracy of EoMT, the state-of-the-art efficient ViT-segmenter, at lower cost, while attaching unchanged to isotropic, hierarchical, convolutional, and mixture-of-experts backbones alike. Holding this head fixed, we benchmark 20 backbones, 11 pretraining schemes and a range of resolutions on ADE20K and Cityscapes under one modern recipe. We find that ``efficient'' token mixers fail to deliver efficiency even at the high resolutions that motivate them, with plain ViT holding the throughput Pareto-front at every resolution. Additionally, the pretraining objective, not the architecture, the lever the field has tuned hardest, governs segmentation quality.
BibTeX
@misc{nauen2026luma,
  author = {Tobias Christian Nauen and Anosh Billimoria and Federico Raue and Stanislav Frolov and Brian B. Moser and Andreas Dengel},
  title = {{LUMA: Benchmarking Segmentation via a Lightweight Universal Mask Adapter}},
  year = {2026},
  eprint = {2607.00687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.00687},
}
Figure