Gated Fully Fusion for Semantic Segmentation

Xiangtai Li, Houlong Zhao, Lei Han, Yunhai Tong, Shaohua Tan, Kuiyuan Yang

Abstract

Semantic segmentation requires a delicate balance between high-level semantic context and low-level spatial precision. Traditional architectures often struggle with the "semantic gap" between these layers, leading to noise or loss of detail. This paper introduces Gated Fully Fusion (GFF), a novel architecture that employs a duplex gating mechanism to selectively fuse features from all network layers simultaneously. By filtering irrelevant information and reducing redundancy, GFF significantly improves segmentation accuracy, particularly for small and thin objects like traffic lights and poles.

Introduction

Deep Convolutional Neural Networks (CNNs) have revolutionized semantic segmentation by enabling pixel-wise category labeling. However, a fundamental challenge remains: high-level features from deeper layers provide rich semantics but lack resolution, while low-level features from shallower layers offer fine spatial details but weak semantic context. While existing solutions like Feature Pyramid Networks (FPNs) attempt to bridge this gap through fusion, simple addition or concatenation often introduces noise, where irrelevant high-level information or context-free low-level details degrade the final prediction.

Methodology

The proposed framework addresses these limitations through two key innovations: Gated Fully Fusion (GFF) and the Dense Feature Pyramid (DFP).

Gated Fully Fusion (GFF)

Unlike standard architectures that fuse only adjacent layers, GFF implements a "fully connected" fusion strategy where every layer can communicate with every other layer. To manage this information flow, a duplex gating mechanism is employed.

The overall architecture of the GFF framework, showing the integration of GFF modules and the Dense Feature Pyramid.

The gating mechanism consists of two primary roles:

  1. The Gate ($G_l$): A learned, pixel-wise mask that evaluates the utility of a specific feature map.
  2. Selective Propagation: Information is only propagated if the "sender" layer contains useful features and the "receiver" layer lacks that specific information. This dual-filtering process prevents the "drowning" of important signals in noise.

Dense Feature Pyramid (DFP)

To complement GFF's focus on fine details, the Dense Feature Pyramid (DFP) handles large-scale objects. By using dense top-down connections, DFP reuses high-level contextual information multiple times, ensuring global consistency across the segmentation map.

Experimental Results

The GFF model was evaluated on four major benchmarks: Cityscapes, ADE20K, Pascal Context, and COCO-stuff.

Performance and Benchmarks

GFF achieved state-of-the-art (SOTA) results at the time of publication:

  • Cityscapes: 82.3% mIoU, outperforming models like DANet and PSANet.
  • ADE20K: 45.33% mIoU.
  • Efficiency: The architecture is remarkably efficient, adding only 7.7% to computational cost (FLOPs) and 6.3% to parameter count compared to a standard PSPNet baseline.

Small Object Mastery

A standout feature of GFF is its ability to segment small, thin objects such as distant traffic lights and poles. Qualitative analysis shows that while previous SOTA models often missed these features entirely, GFF's selective fusion of low-level spatial details allows for precise recovery of these challenging structures.

Qualitative comparison showing GFF's superior ability to segment small objects like traffic lights compared to the PSPNet baseline.

Conclusion

Gated Fully Fusion provides a robust solution to the semantic gap in segmentation. By moving beyond linear, adjacent-layer fusion to a gated, fully connected approach, the model achieves superior accuracy with minimal computational overhead. Future work may involve extending this gating logic to related computer vision tasks like object detection and instance segmentation.

Summary updated

Abstract

Semantic segmentation generates comprehensive understanding of scenes through densely predicting the category for each pixel. High-level features from Deep Convolutional Neural Networks already demonstrate their effectiveness in semantic segmentation tasks, however the coarse resolution of high-level features often leads to inferior results for small/thin objects where detailed information is important. It is natural to consider importing low level features to compensate for the lost detailed information in high-level features. Unfortunately, simply combining multi-level features suffers from the semantic gap among them. In this paper, we propose a new architecture, named Gated Fully Fusion(GFF), to selectively fuse features from multiple levels using gates in a fully connected way. Specifically, features at each level are enhanced by higher-level features with stronger semantics and lower-level features with more details, and gates are used to control the propagation of useful information which significantly reduces the noises during fusion. We achieve the state of the art results on four challenging scene parsing datasets including Cityscapes, Pascal Context, COCO-stuff and ADE20K.
BibTeX
@inproceedings{li2020gated,
  author = {Xiangtai Li and Houlong Zhao and Lei Han and Yunhai Tong and Shaohua Tan and Kuiyuan Yang},
  title = {{Gated Fully Fusion for Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6805},
}
Figure