Introduction
Semantic segmentation has traditionally been framed as a dense prediction task over a regular Cartesian grid. While this structure aligns well with standard convolutional operations, it introduces significant limitations. First, the local nature of CNN receptive fields makes capturing long-range dependencies difficult. Second, real-world objects rarely conform to rigid square grid cells, necessitating complex and heavy decoders to recover fine-grained boundaries from coarse feature maps.
The Region Proxy Approach
RegProxy (Region Proxy) proposes a fundamental shift: instead of operating on pixels, the model identifies and reasons about semantically meaningful regions from the earliest stages of the network. By moving away from rigid grids, the model can more naturally represent the underlying geometry of a scene.
Methodology
The architecture discards the standard 2D feature map layout in favor of a sequence-to-sequence pipeline. The core innovation is the Proxy Head, which consists of two parallel branches:
- Token Branch: This branch generates $N$ region embeddings, or "tokens," representing the semantic content of specific areas.
- Association Branch: This branch learns an "affinity map," assigning each pixel to a nearby token with a certain probability. This map defines the geometric shape of the regions.
These tokens are then processed by a standard Transformer (e.g., ViT). Unlike typical Vision Transformers that treat fixed patches as tokens, RegProxy's tokens represent flexible, learnable regions. This allows the self-attention mechanism to model relationships between actual semantic components of a scene. Finally, a simple linear classifier predicts categories for each token, and the results are "painted" back onto the pixel grid using the learned affinity map. This design eliminates the need for computationally expensive decoders like those found in U-Net or DeepLab.
Experimental Evaluation
RegProxy was evaluated on major benchmarks including ADE20K, Cityscapes, and Pascal Context. The results highlight a superior efficiency-performance trade-off compared to both CNN-based and other Transformer-based architectures.
On the ADE20K dataset, the RegProxy-S/16 variant outperformed top-tier CNN models while utilizing only 25% of the parameters and a mere 4% of the GFLOPs. The largest model, RegProxy-L/16, achieved a state-of-the-art 52.9 mIoU, surpassing previous benchmarks while maintaining a lower computational footprint. On Cityscapes, the model consistently ran at higher frame rates (FPS) than competing Transformer-based models like Segmenter or SETR.
Discussion and Key Insights
The success of RegProxy suggests that semantic homogeneity is a powerful inductive bias. By allowing tokens to represent flexible regions, the model naturally aligns with object boundaries even without explicit geometric regularization. Furthermore, the "no-decoder" advantage demonstrates that if internal representations are sufficiently semantically rich and geometrically aware, heavy upsampling stages become redundant.
However, the model remains dependent on large-scale pre-training (e.g., ImageNet-21k) and currently utilizes fixed $3 \times 3$ neighborhoods for pixel-to-token association. Future work may explore dynamic or global associations to better handle disconnected objects and extend the region proxy concept to instance segmentation and object detection, where region-based reasoning is even more critical.
Summary updated