GSTO: Gated Scale-Transfer Operation for Multi-Scale Feature Learning in Pixel Labeling

Zhuoying Wang, Yongtao Wang, Zhi Tang, Yangyan Li, Ying Chen, Haibin Ling, Weisi LinarXiv 2020 · OpenReview

TL;DR

This paper introduces Gated Scale-Transfer Operations (GSTO), a novel mechanism to improve how neural networks learn multi-scale features for pixel labeling tasks like semantic segmentation and human pose estimation. By introducing a "gating" mechanism that selectively transfers features across scales, GSTO addresses the issue of "scale-confused" features in existing methods, leading to state-of-the-art results with minimal computational overhead.

What Problem Does This Paper Address?

Pixel labeling tasks, such as semantic segmentation (assigning a category to each pixel, like "road," "car," "person") and human pose estimation (identifying keypoints of human bodies), require models to understand both the broad context of an image (semantic comprehension) and the fine details (localization). Current approaches often rely on processing features at multiple scales (e.g., using upsampling and downsampling) to achieve this. However, the paper argues that these conventional scale-transfer operations are "spatial-invariant," meaning they treat all parts of a feature map equally when moving between scales. This can lead to "scale-confused" features, where information from different scales gets mixed inappropriately, hindering performance. For instance, fine details might be lost on low-resolution maps, or large objects might not be well-represented on high-resolution maps. The core problem is how to effectively transfer and integrate multi-scale features in a way that preserves both semantic understanding and spatial precision.

Methods

The paper proposes Gated Scale-Transfer Operations (GSTO), a novel module designed to improve cross-scale feature transfer. The key idea is to introduce a "spatial mask" or "gate" that selectively filters features before they are transferred to another scale. This gate learns which features are relevant for the target scale and which are not, thus preventing scale confusion.

GSTO comes in two flavors:

  • Unsupervised GSTO: This version learns the gating map directly from the feature map itself. It uses a 1x1 convolution followed by a sigmoid activation to generate the spatial gate. This makes it lightweight and efficient.
  • Supervised GSTO: This version uses additional supervision during training. It leverages the probability map of semantic categories to guide the learning of the spatial mask. This can lead to more accurate gating but requires extra supervision.

Both GSTO variants are designed to be "plug-and-play," meaning they can be easily integrated into existing neural network architectures without significant modifications. The authors demonstrate this by incorporating GSTO into the HRNet backbone, creating GSTO-HRNet, and also by applying it to multi-scale aggregation modules like Pyramid Pooling Module (PPM) and Atrous Spatial Pyramid Pooling (ASPP).

Datasets

The paper evaluates its proposed methods on several standard benchmarks for pixel labeling tasks:

  • Semantic Segmentation:
    • Cityscapes
    • LIP (Human Parsing)
    • Pascal Context
  • Pose Estimation:
    • COCO

No new datasets were introduced.

Key Insights and Ideas

  • Scale Confusion is a Real Problem: The paper effectively demonstrates that standard up/downsampling operations in multi-scale feature learning are too simplistic and lead to features that are not truly "scale-aware," causing confusion between fine and coarse details.
  • Gating is Key to Scale-Awareness: Introducing a learned spatial gate allows the network to selectively pass information across scales, ensuring that features relevant to a specific scale are emphasized.
  • Lightweight and Plug-and-Play: The proposed GSTO modules are designed to be computationally efficient and easily integrated into existing architectures, making them practical for real-world applications.
  • Unsupervised Learning is Powerful: The unsupervised version of GSTO shows significant improvements without requiring additional labeled data, highlighting its efficiency and effectiveness.
  • HRNet is a Strong Foundation: The paper leverages the powerful HRNet architecture and shows that GSTO can further enhance its already impressive performance.

Key Results

The GSTO-based methods achieved state-of-the-art (SOTA) results on multiple benchmarks:

  • Semantic Segmentation (Cityscapes): GSTO-HRNet achieved 81.8% mIoU (using only fine-labeled training data) and 82.4% mIoU (using fine-labeled train-val data), outperforming previous SOTA. This was achieved with only a marginal increase in parameters (less than 1%) and computational cost (less than 5%).
  • Human Pose Estimation (COCO): GSTO-HRNet-W32 improved AP from 75.8% to 76.5%, and GSTO-HRNet-W48 reached 76.7% AP, outperforming the original HRNet-W48 with fewer parameters and lower computational cost.
  • Multi-scale Modules: Applying GSTO to PPM and ASPP modules also led to significant improvements, with supervised GSTO boosting PPM by 1.3% and ASPP by 2.0% on Cityscapes.
  • Qualitative Results: Visualizations (Figure 5) show that GSTO-HRNet produces more accurate boundary details, better consistency on large objects, and improved semantic comprehension compared to the baseline HRNet.

Limitations and Future Work

  • Supervised GSTO's Reliance on Supervision: While supervised GSTO can be effective, its performance is tied to the quality and availability of semantic labels for training the gating mechanism. The paper notes that unsupervised GSTO still provides substantial gains, suggesting its practicality.
  • Number of Stages for Supervised GTM: The paper explored the number of stages for supervised Gated Transition Modules (GTM). They found that using supervised GTM in only stages 2 and 3 (rather than all stages) yielded the best results, suggesting that applying it too early in shallow layers might be detrimental. This indicates a need for careful consideration of where to apply supervised components.
  • Future Work: The paper implicitly suggests that further exploration of different gating mechanisms and their integration into various network architectures could be beneficial. The success of GSTO in enhancing both backbone networks and feature aggregation modules points towards its broad applicability in computer vision tasks requiring multi-scale feature understanding.

Summary updated

Abstract

Existing CNN-based methods for pixel labeling heavily depend on multi-scale features to meet the requirements of both semantic comprehension and detail preservation. State-of-the-art pixel labeling neural networks widely exploit conventional scale-transfer operations, i.e., up-sampling and down-sampling to learn multi-scale features. In this work, we find that these operations lead to scale-confused features and suboptimal performance because they are spatial-invariant and directly transit all feature information cross scales without spatial selection. To address this issue, we propose the Gated Scale-Transfer Operation (GSTO) to properly transit spatial-filtered features to another scale. Specifically, GSTO can work either with or without extra supervision. Unsupervised GSTO is learned from the feature itself while the supervised one is guided by the supervised probability matrix. Both forms of GSTO are lightweight and plug-and-play, which can be flexibly integrated into networks or modules for learning better multi-scale features. In particular, by plugging GSTO into HRNet, we get a more powerful backbone (namely GSTO-HRNet) for pixel labeling, and it achieves new state-of-the-art results on the COCO benchmark for human pose estimation and other benchmarks for semantic segmentation including Cityscapes, LIP and Pascal Context, with negligible extra computational cost. Moreover, experiment results demonstrate that GSTO can also significantly boost the performance of multi-scale feature aggregation modules like PPM and ASPP. Code will be made available at https://github.com/VDIGPKU/GSTO.
BibTeX
@misc{wang2020gsto,
  author = {Zhuoying Wang and Yongtao Wang and Zhi Tang and Yangyan Li and Ying Chen and Haibin Ling and Weisi Lin},
  title = {{GSTO: Gated Scale-Transfer Operation for Multi-Scale Feature Learning in Pixel Labeling}},
  year = {2020},
  eprint = {2005.13363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.13363},
}
Figure