TL;DR
Evaluating vision backbones for image segmentation has long been an unfair game, as each model is typically paired with a different decoder and training recipe. This paper introduces LUMA (Lightweight Universal Mask Adapter), a minimal, backbone-agnostic head that acts as a fair "measuring stick" to benchmark 20 transformer architectures and 11 pretraining schemes. The surprising verdict? Plain ViT remains the efficiency king, and the pretraining objective matters far more than clever architectural tweaks.
What Problem Does This Paper Address?
In computer vision, designing "efficient" token mixers (like Swin, Linformer, or specialized convolutional hybrids) is a popular research direction aimed at avoiding the quadratic cost of standard attention at high resolutions. However, when these backbones are evaluated on downstream tasks like semantic segmentation, they are bundled with heavy, task-specific decoders (e.g., Mask2Former) and custom training recipes. This confounds the results: is a high mIoU score due to a superior backbone, a massive decoder, or a lucky pretraining setup?
Previous attempts to strip away the decoder clutter, such as EoMT, only worked on plain, isotropic ViTs because they embedded segmentation queries directly into the backbone's global self-attention mechanism. This paper addresses the need for a universal, lightweight, and non-intrusive segmentation head that can attach to any backbone architecture—isotropic, hierarchical, convolutional, or mixture-of-experts—allowing researchers to isolate and fairly evaluate the backbone's true capabilities.
Methods
The authors introduce the Lightweight Universal Mask Adapter (LUMA). Instead of forcing segmentation queries into the backbone's internal token stream, LUMA keeps them in a separate side-stream.
Key methodological highlights include:
- Black-Box Interaction: LUMA reads features from the backbone's blocks using a single, cheap cross-attention layer per tapped block. The backbone runs exactly as originally designed.
- Weight and Schedule Reuse: After cross-attention, the queries are processed using the backbone's own existing Feed-Forward Network (MLP) layers. This means queries automatically inherit the backbone's width and dimension schedule without requiring redundant, architecture-specific projection layers.
- Unmasked Simplicity: Unlike prior work that relies on complex masked-attention and annealing schedules during training, LUMA achieves state-of-the-art accuracy using plain, unmasked cross-attention, significantly simplifying the training pipeline.
Datasets
The unified benchmark evaluates the models on two standard segmentation datasets:
- ADE20K
- Cityscapes
Using LUMA as the fixed head, the authors benchmarked 20 distinct backbone architectures, 11 pretraining schemes, and multiple image resolutions under a single modern training recipe.
Key Insights and Ideas
- ImageNet Accuracy is a Poor Proxy: A backbone's top-1 accuracy on ImageNet-1k is an unreliable predictor of its final segmentation performance. Many specialized or convolutional hybrid models fall well below the expected trend line when moving to dense prediction tasks.
- The "Efficient" Token Mixer Illusion: Many architectures designed specifically to be "efficient" at high resolutions fail to deliver actual throughput benefits. When evaluated fairly, plain ViT (especially when boosted by FlashAttention) consistently holds the Pareto-front for speed and accuracy. Specialized mixers often only offer advantages in memory consumption, not speed.
- Pretraining Trumps Architecture: The pretraining objective is the strongest lever for segmentation quality. Dense self-supervised objectives (like DINOv2/v3 or masked image modeling via EVA-02) transfer robustly to dense tasks, whereas standard supervised pretraining (like DeiT III) performs poorly, sitting up to 8 percentage points lower in mIoU.
- The 80/20 Rule for Scaling: When scaling up a compute budget, the optimal path is highly asymmetric. The authors' compute-optimal scaling analysis shows that roughly 80% of additional compute should be allocated to increasing model parameter size, while only 20% should go toward increasing input resolution.
Key Results
- State-of-the-Art Efficiency: Paired with a canonical ViT-L/DINOv2 backbone, LUMA achieves 84.4 mIoU on Cityscapes and 58.7 mIoU on ADE20K. This matches or exceeds heavy decoder baselines while using 1.3× fewer GFLOPs and running over 7× faster than a ViT-Adapter + Mask2Former setup.
- High-Resolution Collapse: Several sequence-length-bound mixers fail to generalize to the high resolutions required for segmentation. For instance, Linformer's performance collapses from 38.4 mIoU down to 3.8 mIoU as resolution scales up, a critical flaw that classification benchmarks completely hide.
Limitations and Future Work
- Pretraining Availability Confound: While the paper concludes that dense pretraining is superior, dense-objective pretrained weights are currently only widely available for plain ViT architectures. Consequently, the broad 20-backbone architectural comparison had to rely on a shared supervised pretraining setup.
- Resolution Saturation: The benefits of resolution scaling are strictly bounded by the native resolution of the dataset's images, meaning resolution scaling eventually hits a hard ceiling.
- Dataset Quirks: Scaling behaviors can be dataset-dependent; for example, the massive ViT-g model proved compute-optimal on the challenging ADE20K dataset but was suboptimal on Cityscapes compared to smaller models running at higher resolutions.
Summary updated