Semantic Segmentation by Early Region Proxy

Yifan Zhang, Bo Pang, Cewu LuCVPR 2022 · arXiv

Introduction

Semantic segmentation has traditionally been framed as a dense prediction task over a regular Cartesian grid. While this structure aligns well with standard convolutional operations, it introduces significant limitations. First, the local nature of CNN receptive fields makes capturing long-range dependencies difficult. Second, real-world objects rarely conform to rigid square grid cells, necessitating complex and heavy decoders to recover fine-grained boundaries from coarse feature maps.

The Region Proxy Approach

RegProxy (Region Proxy) proposes a fundamental shift: instead of operating on pixels, the model identifies and reasons about semantically meaningful regions from the earliest stages of the network. By moving away from rigid grids, the model can more naturally represent the underlying geometry of a scene.

Conceptual comparison between traditional dense correspondence and the proposed region-wise correspondence.

Methodology

The architecture discards the standard 2D feature map layout in favor of a sequence-to-sequence pipeline. The core innovation is the Proxy Head, which consists of two parallel branches:

  1. Token Branch: This branch generates $N$ region embeddings, or "tokens," representing the semantic content of specific areas.
  2. Association Branch: This branch learns an "affinity map," assigning each pixel to a nearby token with a certain probability. This map defines the geometric shape of the regions.

These tokens are then processed by a standard Transformer (e.g., ViT). Unlike typical Vision Transformers that treat fixed patches as tokens, RegProxy's tokens represent flexible, learnable regions. This allows the self-attention mechanism to model relationships between actual semantic components of a scene. Finally, a simple linear classifier predicts categories for each token, and the results are "painted" back onto the pixel grid using the learned affinity map. This design eliminates the need for computationally expensive decoders like those found in U-Net or DeepLab.

Experimental Evaluation

RegProxy was evaluated on major benchmarks including ADE20K, Cityscapes, and Pascal Context. The results highlight a superior efficiency-performance trade-off compared to both CNN-based and other Transformer-based architectures.

Performance comparison (mIoU vs. GFLOPs) on the ADE20K dataset, showing RegProxy's efficiency advantage over existing architectures.

On the ADE20K dataset, the RegProxy-S/16 variant outperformed top-tier CNN models while utilizing only 25% of the parameters and a mere 4% of the GFLOPs. The largest model, RegProxy-L/16, achieved a state-of-the-art 52.9 mIoU, surpassing previous benchmarks while maintaining a lower computational footprint. On Cityscapes, the model consistently ran at higher frame rates (FPS) than competing Transformer-based models like Segmenter or SETR.

Discussion and Key Insights

The success of RegProxy suggests that semantic homogeneity is a powerful inductive bias. By allowing tokens to represent flexible regions, the model naturally aligns with object boundaries even without explicit geometric regularization. Furthermore, the "no-decoder" advantage demonstrates that if internal representations are sufficiently semantically rich and geometrically aware, heavy upsampling stages become redundant.

However, the model remains dependent on large-scale pre-training (e.g., ImageNet-21k) and currently utilizes fixed $3 \times 3$ neighborhoods for pixel-to-token association. Future work may explore dynamic or global associations to better handle disconnected objects and extend the region proxy concept to instance segmentation and object detection, where region-based reasoning is even more critical.

Summary updated

Abstract

Typical vision backbones manipulate structured features. As a compromise, semantic segmentation has long been modeled as per-point prediction on dense regular grids. In this work, we present a novel and efficient modeling that starts from interpreting the image as a tessellation of learnable regions, each of which has flexible geometrics and carries homogeneous semantics. To model region-wise context, we exploit Transformer to encode regions in a sequence-to-sequence manner by applying multi-layer self-attention on the region embeddings, which serve as proxies of specific regions. Semantic segmentation is now carried out as per-region prediction on top of the encoded region embeddings using a single linear classifier, where a decoder is no longer needed. The proposed RegProxy model discards the common Cartesian feature layout and operates purely at region level. Hence, it exhibits the most competitive performance-efficiency trade-off compared with the conventional dense prediction methods. For example, on ADE20K, the small-sized RegProxy-S/16 outperforms the best CNN model using 25% parameters and 4% computation, while the largest RegProxy-L/16 achieves 52.9mIoU which outperforms the state-of-the-art by 2.1% with fewer resources. Codes and models are available at https://github.com/YiF-Zhang/RegionProxy.
BibTeX
@inproceedings{zhang2022semantic,
  author = {Yifan Zhang and Bo Pang and Cewu Lu},
  title = {{Semantic Segmentation by Early Region Proxy}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure