Global Aggregation then Local Distribution for Scene Parsing

Xiangtai Li, Li Zhang, Guangliang Cheng, Kuiyuan Yang, Yunhai Tong, Xiatian Zhu, Tao Xiang

Abstract

Modelling long-range contextual relationships is critical for pixel-wise prediction tasks such as semantic segmentation. However, convolutional neural networks (CNNs) are inherently limited in modelling such dependencies due to the local nature of convolution kernels. While recent global aggregation (GA) methods improve long-range structure modelling, they often over-smooth features, introducing noise to regions containing fine details, such as boundaries and small objects. To alleviate this, we propose the Global Aggregation then Local Distribution (GALD) framework. By exploring local context, GALD ensures that aggregated long-range relationships are distributed accurately to local regions. We introduce a novel local distribution module that adaptively models the affinity map between global and local relationships for each pixel. Our approach is modular, end-to-end trainable, and easily integrated into existing networks. GALD achieves state-of-the-art performance on major benchmarks, including Cityscapes, ADE20K, Pascal Context, Camvid, and COCO-stuff.

Introduction

Semantic segmentation, or scene parsing, requires a model to assign a semantic category to every pixel in an image. A significant challenge in this field is the tension between global context and local precision. Standard CNNs possess limited receptive fields, making it difficult for them to distinguish between classes that look similar locally but differ globally (e.g., a patch of gray pixels representing a road versus a building wall).

To address this, researchers have developed Global Aggregation (GA) modules—such as Atrous Spatial Pyramid Pooling (ASPP), Pyramid Pooling Modules (PSP), or Non-local blocks—to capture scene-wide statistics. However, these modules often suffer from "over-smoothing." Because they aggregate information across the entire image, the resulting features are frequently biased toward dominant, large-scale objects. When this global context is broadcast back to the pixel level, it can overwhelm the unique features of small objects and blur the sharp boundaries between categories.

The GALD Framework

The GALD framework addresses the over-smoothing problem by decoupling the contextual process into two distinct stages: Global Aggregation and Local Distribution (LD). While the GA module gathers the necessary long-range dependencies, the LD module acts as a smart gatekeeper, determining how that global information should be distributed to individual pixels based on their local environment.

Evolution of the Local Distribution Module

The paper explores two iterations of the distribution logic:

  1. LDv1: This version utilizes depth-wise convolutions to estimate point-wise masks on the GA features. It employs residual learning to sharpen the representation of large objects while attempting to preserve the integrity of smaller ones.
  2. LDv2 (Dynamic Local Distribution): The core contribution of this work is a more sophisticated, attention-based distribution module. Traditional self-attention mechanisms (like Non-local networks) compute an affinity matrix between every pair of pixels, leading to a prohibitive $O(N^2)$ computational complexity. LDv2 introduces a sparse global-to-local graph. For every pixel, the module samples a local $k \times k$ neighborhood using a dilated uniform sampling strategy. By calculating attention weights conditionally based on the GA features within this restricted neighborhood, the complexity is reduced to $O(N \times K)$. This allows the model to achieve the benefits of dense attention with significantly higher efficiency.

Experimental Results

The GALD framework was evaluated across five major semantic segmentation benchmarks, demonstrating consistent improvements over baseline architectures.

State-of-the-Art Performance

On the Cityscapes dataset, GALDv2 achieved a 83.2% mIoU on the test set using a ResNet-101 backbone and fine annotations. When supplemented with coarse data, the performance reached 83.5% mIoU, surpassing previous leading models like Auto-DeepLab. Similar successes were mirrored on other datasets:

  • ADE20K: 46.38% mIoU
  • Pascal Context: 55.2% mIoU
  • COCO-stuff: 40.1% mIoU
  • Camvid: 81.6% mIoU

Boundary Preservation and Efficiency

A key metric for GALD is its performance on object boundaries. Using the F-score to evaluate mask edges, GALD outperformed strong baselines like DeepLabv3+ and G-SCNN by 3% to 4%. This confirms that the LD module successfully prevents the global context from washing out high-frequency edge details.

Furthermore, the module is highly efficient. Integrating LDv2 into an ASPP-based network increased inference time by only 17ms (from 65ms to 82ms) while providing a substantial 3.6 percentage point boost in mIoU.

Conclusion

The GALD framework demonstrates that the primary flaw in current global context models is not how they gather information, but how they distribute it. By treating distribution as an adaptive, local process, GALD preserves fine details and sharp boundaries that are often lost in traditional global aggregation. Its architecture-agnostic, plug-and-play nature makes it a versatile tool for enhancing a wide variety of existing semantic segmentation networks. Future research may extend this "aggregate-then-distribute" philosophy to other dense prediction tasks such as depth estimation and panoptic segmentation.

Summary updated

Abstract

Modelling long-range contextual relationships is critical for pixel-wise prediction tasks such as semantic segmentation. However, convolutional neural networks (CNNs) are inherently limited to model such dependencies due to the naive structure in its building modules (\eg, local convolution kernel). While recent global aggregation methods are beneficial for long-range structure information modelling, they would oversmooth and bring noise to the regions containing fine details (\eg,~boundaries and small objects), which are very much cared for the semantic segmentation task. To alleviate this problem, we propose to explore the local context for making the aggregated long-range relationship being distributed more accurately in local regions. In particular, we design a novel local distribution module which models the affinity map between global and local relationship for each pixel adaptively. Integrating existing global aggregation modules, we show that our approach can be modularized as an end-to-end trainable block and easily plugged into existing semantic segmentation networks, giving rise to the \emph{GALD} networks. Despite its simplicity and versatility, our approach allows us to build new state of the art on major semantic segmentation benchmarks including Cityscapes, ADE20K, Pascal Context, Camvid and COCO-stuff. Code and trained models are released at \url{https://github.com/lxtGH/GALD-DGCNet} to foster further research.
BibTeX
@article{li2021global,
  author = {Xiangtai Li and Li Zhang and Guangliang Cheng and Kuiyuan Yang and Yunhai Tong and Xiatian Zhu and Tao Xiang},
  title = {{Global Aggregation then Local Distribution for Scene Parsing}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  doi = {10.1109/tip.2021.3099366},
}
Figure