Distributed Iterative Gating Networks for Semantic Segmentation

Rezaul Karim, Md Amirul Islam, Neil D. B. BruceWACV 2020

TL;DR

This paper introduces DIGNet, a novel recurrent architecture for semantic segmentation that uses a cascaded feedback mechanism to iteratively refine predictions. By allowing information to flow top-down from deeper layers to earlier ones, DIGNet effectively bridges the gap between high-level semantic understanding and low-level spatial details, leading to more accurate and robust segmentation.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, has seen significant advancements with deep learning. However, a key challenge remains: balancing the rich semantic information captured by deep layers with the precise spatial details needed for accurate pixel-level labeling. Deeper networks, while good at understanding "what" is in an image, often lose spatial resolution, making it difficult to delineate object boundaries accurately. Conversely, shallower layers retain spatial detail but lack semantic context. Existing methods often struggle to effectively integrate these two aspects. This paper addresses this by proposing a mechanism that allows deeper semantic features to iteratively guide and refine the representations in earlier layers, thereby improving both spatial accuracy and semantic understanding.

Methods

The core innovation of DIGNet is its Distributed Iterative Gating (DIGNet) mechanism. This architecture augments a standard feed-forward convolutional neural network (CNN) backbone with two types of gating modules:

  • Propagator Gates (Gp): These modules are responsible for generating feedback signals. They operate in a top-down manner, taking feedback from the layer above and the current bottom-up features to create a new, refined feedback signal. This cascaded approach ensures that information from all deeper layers is implicitly carried to earlier stages.
  • Modulator Gates (Gm): These gates are placed between intermediate stages of the feed-forward backbone. They use the feedback signals generated by the propagator gates to modulate (e.g., re-weight) the features in the feed-forward path. This modulation helps to refine the intermediate representations based on higher-level semantic context.

DIGNet operates iteratively. In the first iteration, it functions like a standard feed-forward network. In subsequent iterations, the feedback mechanism is activated, allowing for the refinement of features. This iterative process is trained using back-propagation through time (BPTT), with the loss computed only on the final output.

Datasets

The paper evaluates DIGNet on three widely used semantic segmentation datasets:

  • PASCAL VOC 2012: A standard benchmark for semantic segmentation.
  • ADE20K: A more complex dataset with a larger number of classes and images.
  • COCO-Stuff10k: A dataset focused on scene parsing with a large number of object and stuff categories.

Key Insights and Ideas

  • Cascaded Feedback is Key: The paper argues that a cascaded feedback mechanism, where information flows from deeper to shallower layers in a structured way, is more effective than simply using feedback from the last layer or stage-wise feedback. This allows for a more nuanced and comprehensive guidance of earlier features.
  • Bridging the Semantic-Spatial Gap: DIGNet's iterative refinement process effectively bridges the gap between high-level semantic understanding and low-level spatial details. This is crucial for tasks requiring precise object localization and boundary delineation.
  • Efficient Iterative Refinement: The iterative nature of DIGNet allows for fast convergence, achieving significant performance gains with only a few iterations (T=2 was used in experiments). This suggests that DIGNet offers a good trade-off between performance and computational cost.
  • Implicit Hypercolumn Representation: The cascaded feedback mechanism can be viewed as creating a compact hypercolumn representation, where each stage receives guidance from all subsequent processing stages.

Key Results

DIGNet consistently outperforms its feed-forward baselines and other recurrent feedback methods across the evaluated datasets.

  • PASCAL VOC 2012: DIGNet achieved state-of-the-art results, with ResNet101-DIGNet (T=2) reaching 77.5% mIoU on the validation set and 80.7% mIoU on the test set, outperforming strong baselines like DeepLabv2.
  • ADE20K: DIGNet demonstrated significant improvements, with ResNet101(8s)-DIGNet achieving 36.9% mIoU, outperforming baselines by a notable margin.
  • COCO-Stuff10k: DIGNet provided substantial performance boosts, with ResNet101(8s)-DIGNet achieving 36.3% mIoU, a significant improvement over its baseline.
  • Qualitative Results: Visual comparisons showed DIGNet's ability to resolve categorical ambiguity, improve partial segmentation, recover missing small objects, and refine coarse-grained representations.

Limitations and Future Work

  • Failure Cases: While DIGNet performs well, there are rare cases where objects with very similar visual features to the background can lead to incorrect labeling, even if the overall prediction remains globally consistent. This suggests that in extreme cases of visual ambiguity, DIGNet might still struggle.
  • Future Work: The authors suggest that DIGNet's iterative inference and cascaded feedback mechanism hold promise for improving semantic segmentation in challenging scenarios. Further exploration of this feedback-guided iterative inference mechanism is a natural next step. The paper also hints at the potential for DIGNet to be applied to other vision tasks beyond semantic segmentation.

Summary updated

BibTeX
@inproceedings{karim2020distributed,
  author = {Rezaul Karim and Md Amirul Islam and Neil D. B. Bruce},
  title = {{Distributed Iterative Gating Networks for Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure