Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation

Loic Themyr, Clément Rambour, Nicolas Thome, Toby Collins, Alexandre Hostettler

Introduction

Semantic segmentation—the process of assigning a category label to every pixel in an image—is a cornerstone of computer vision applications ranging from autonomous driving to medical diagnostics. While Vision Transformers (ViTs) have set new benchmarks in this field, they encounter a significant "resolution wall." The standard self-attention mechanism possesses quadratic complexity relative to image resolution, making high-resolution processing computationally prohibitive.

To mitigate this, architectures like the Swin Transformer employ window-based attention, restricting calculations to local sub-windows. However, this efficiency comes at the cost of "tunnel vision," where the model loses the ability to capture long-range dependencies across the image. The GLAM (GLobal Attention Multi-resolution) module addresses this by restoring global context without sacrificing the efficiency of window-based approaches.

The GLAM Module: Bridging Local and Global Context

The core innovation of GLAM is the introduction of learnable "global tokens" that act as intermediaries between isolated local windows. The process follows a structured two-step attention mechanism:

  1. Local Interaction (W-MSA): Within each window, visual patches interact with a set of global tokens. These tokens effectively "absorb" and summarize the local information.
  2. Global Interaction (G-MSA): These global tokens then interact across all windows, allowing information to propagate throughout the entire image.
Semantic segmentation overlay on a Cityscapes urban driving scene, demonstrating the model's ability to handle complex spatial relationships.

Furthermore, the authors introduce Non-Local Upsampling (NLU) in the decoder stage. Unlike standard interpolation, NLU utilizes an attention mechanism to upsample features based on the global context aggregated during encoding, ensuring that fine-grained details are reconstructed with global awareness.

Experimental Evaluation

The GLAM module was evaluated across three distinct benchmarks: ADE20K (general scenes), Cityscapes (urban driving), and Synapse (medical CT scans).

Performance on Urban and General Scenes

On the Cityscapes dataset, the integration of GLAM resulted in a +1.58% improvement over the Swin-UperNet baseline. Similarly, on ADE20K, the GLAM-Swin-Unet achieved 49.10% mIoU, a 1.25-point increase over the standard architecture. These results highlight the module's ability to resolve ambiguities in complex environments by leveraging global priors.

A detailed segmentation mask illustrating the precise object boundaries achieved through global contextual awareness.

State-of-the-Art in Medical Imaging

In the medical domain, GLAM-nnFormer set a new state-of-the-art on the Synapse dataset with an average Dice Similarity Score (DSC) of 88.60%, outperforming the previous best by 1.2 points. This suggests that global context is particularly vital for identifying anatomical structures in 3D volumetric data.

Conclusion and Future Work

The GLAM module demonstrates that global context can be efficiently reintegrated into hierarchical transformers. By using global tokens as "messengers," the architecture achieves a superior performance-per-parameter ratio compared to simply scaling up model size. Future research may explore the application of GLAM to ultra-high-resolution imagery and its optimization for 3D volumetric tasks where computational costs remain a challenge.

Summary updated

Abstract

Transformers have proved to be very effective for visual recognition tasks. In particular, vision transformers construct compressed global representations through self-attention and learnable class tokens. Multi-resolution transformers have shown recent successes in semantic segmentation but can only capture local interactions in high-resolution feature maps. This paper extends the notion of global tokens to build GLobal Attention Multi-resolution (GLAM) transformers. GLAM is a generic module that can be integrated into most existing transformer backbones. GLAM includes learnable global tokens, which unlike previous methods can model interactions between all image regions, and extracts powerful representations during training. Extensive experiments show that GLAM-Swin or GLAM-Swin-Unet exhibit substantially better performances than their vanilla counterparts on ADE20K and Cityscapes. Moreover, GLAM can be used to segment large 3D medical images, and GLAM-nnFormer achieves new state-of-the-art performance on the BCV dataset.
BibTeX
@inproceedings{themyr2023full,
  author = {Loic Themyr and Clément Rambour and Nicolas Thome and Toby Collins and Alexandre Hostettler},
  title = {{Full Contextual Attention for Multi-resolution Transformers in Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00324},
}
Figure