Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation

Abhishek Aich, Yumin Suh, Samuel Schulter, Manmohan Chandraker

TL;DR

PRO-SCALE is a "plug-and-play" strategy that slashes the computational cost of state-of-the-art universal segmentation models like Mask2Former. By progressively increasing the number of tokens processed across encoder layers and simplifying pixel embeddings, it reduces encoder GFLOPs by ~52% with virtually no loss in segmentation accuracy.

What Problem Does This Paper Address?

Universal segmentation models (which handle semantic, instance, and panoptic segmentation in one go) have become incredibly powerful, but they are notoriously compute-heavy. The authors identify a major bottleneck: in the popular Mask2Former (M2F) architecture, the transformer encoder consumes over 50% of the total computation.

The transformer encoder is the primary computational bottleneck in M2F-style architectures.

This inefficiency stems from how M2F handles multi-scale image features. It flattens all feature scales (from low-resolution semantic maps to high-resolution detail maps) into a single long sequence of tokens and processes this entire sequence at every single encoder layer. This "one-size-fits-all" approach ignores the fact that high-resolution tokens are often redundant in early processing stages.

Methods

The authors propose PRO-SCALE (PROgressive Token Length SCALing), which introduces three key innovations:

  1. Progressive Token Expansion: Instead of feeding all tokens into every layer, PRO-SCALE starts with only the coarsest, most semantically rich tokens ($s_4$). As the data moves deeper into the encoder, it progressively adds higher-resolution tokens ($s_3$ then $s_2$). This "semantic-first" approach focuses compute where it matters most early on.
  2. Light Pixel Embedding (LPE): Standard M2F uses heavy convolutional layers to create the pixel embeddings used for final mask prediction. PRO-SCALE replaces these with a parameter-free module using simple max-pooling and normalization, significantly reducing overhead.
  3. Token Re-Calibration (TRC): To ensure that skipping high-res tokens early doesn't lead to a loss of detail, the TRC module uses high-resolution features to "calibrate" the lower-resolution tokens via a lightweight attention mechanism before they enter the encoder.
The PRO-SCALE framework progressively expands token length and utilizes a Light Pixel Embedding (LPE) module.

Datasets

The researchers evaluated PRO-SCALE on three major benchmarks:

  • COCO: The gold standard for general object detection and segmentation.
  • Cityscapes: A dataset focused on urban street scenes, critical for autonomous driving research.
  • ADE20K: Used for cross-dataset evaluation to test the model's robustness.

Key Insights and Ideas

  • High-Res Redundancy: The authors found that neighboring high-resolution tokens have extremely high similarity in early layers (see Figure 9 in the paper). Processing them all immediately is a waste of GFLOPs.
  • Efficiency without Sacrifice: Unlike previous "efficient" versions of DETR-style models that often sacrifice accuracy for speed, PRO-SCALE maintains (and sometimes exceeds) the performance of the original heavy models.
  • Versatility: While designed for segmentation, the authors demonstrated that PRO-SCALE also improves the efficiency of object detection models (like DINO) and open-vocabulary models (like FCCLIP).
Comparison of PRO-SCALE's progressive approach versus prior static encoders.

Key Results

  • Massive Speedup: PRO-SCALE achieved a ~52% reduction in encoder GFLOPs and a ~27% overall reduction in total model GFLOPs.
  • Accuracy Gains: On the COCO dataset, PRO-SCALE with a SWIN-T backbone reached 52.82% PQ (Panoptic Quality), actually outperforming the original Mask2Former's 52.03% PQ while being significantly faster.
  • Qualitative Excellence: Visualizations show that PRO-SCALE maintains sharp boundaries and correctly identifies complex objects even with the reduced computational budget.
Qualitative comparison showing PRO-SCALE maintaining high-quality segmentation maps despite lower compute.

Limitations and Future Work

The authors acknowledge that while PRO-SCALE is highly effective, the specific "scaling factors" (how many layers process which scales) are currently chosen manually. A promising future direction would be to automate the search for these configurations using Neural Architecture Search (NAS) to find even more optimal trade-offs for specific hardware. Additionally, while it works well for M2F-style models, its applicability to non-transformer-based architectures remains to be explored.

Summary updated

Abstract

A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions. With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses 50% of its compute only on the transformer encoder. This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer. With this observation, we propose a strategy termed PROgressive Token Length SCALing for Efficient transformer encoders (PRO-SCALE) that can be plugged-in to the Mask2Former segmentation architecture to significantly reduce the computational cost. The underlying principle of PRO-SCALE is: progressively scale the length of the tokens with the layers of the encoder. This allows PRO-SCALE to reduce computations by a large margin with minimal sacrifice in performance (~52% encoder and ~27% overall GFLOPs reduction with no drop in performance on COCO dataset). Experiments conducted on public benchmarks demonstrates PRO-SCALE's flexibility in architectural configurations, and exhibits potential for extension beyond the settings of segmentation tasks to encompass object detection. Code here: https://github.com/abhishekaich27/proscale-pytorch
BibTeX
@inproceedings{aich2025progressive,
  author = {Abhishek Aich and Yumin Suh and Samuel Schulter and Manmohan Chandraker},
  title = {{Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=dmzM5UdAq6},
}
Figure