Towards Efficient Scene Understanding via Squeeze Reasoning

Xiangtai Li, Ansheng You, Li Zhang, Guangliang Cheng, Kuiyuan Yang, Yunhai Tong, Zhouchen LinIEEE TIP 2020 · DOI · OpenReview

Abstract

Graph-based convolutional models, such as non-local blocks, are effective for strengthening context modeling in convolutional neural networks (CNNs). However, their pixel-wise computational overhead is prohibitive for high-resolution imagery. We propose Squeeze Reasoning (SR), a framework that shifts reasoning from the spatial map to a channel-wise global vector. By building a node graph where each node represents an abstract semantic concept, we significantly reduce computational costs while maintaining consistency within semantic categories. Our approach is modular, end-to-end trainable, and achieves state-of-the-art results across semantic segmentation, object detection, and panoptic segmentation tasks with minimal overhead.

Introduction: The Context Problem

In computer vision, "context" is essential for scene understanding—knowing how a car relates to the road or how a sidewalk relates to a building. Standard Convolutional Neural Networks (CNNs) have a limited receptive field, often failing to capture long-range dependencies. While "Non-local" blocks or self-attention mechanisms address this by allowing every pixel to interact, they suffer from a computational cost that grows quadratically ($O(N^2)$) with the number of pixels. For high-resolution images common in autonomous driving, this overhead is prohibitive. The authors propose Squeeze Reasoning to capture global context without the pixel-wise price tag.

Methodology: Squeeze Reasoning

The Squeeze Reasoning (SR) module consists of three main steps designed to abstract spatial information into semantic nodes:

  1. Node Squeezing: Instead of looking at every pixel, the model "squeezes" the entire spatial feature map into a single global vector. This can be achieved via standard Global Average Pooling (GAP) or Global Hadamard Pooling (GHP), which captures second-order statistics to better preserve complex relations.
  2. Node Graph Reasoning: The squeezed vector is divided into $K$ groups, where each group acts as a "node" representing an abstract semantic concept (e.g., "vegetation" or "vehicles"). The model then uses a Graph Convolutional Network (GCN) or a self-attention mechanism to perform reasoning between these nodes. Because there are only a few nodes (e.g., 16 or 32) compared to thousands of pixels, this step is incredibly efficient.
  3. Feature Reconstruction: The refined, "reasoned" nodes are projected back and used to re-weight the original feature map, effectively broadcasting global insights back to the spatial pixels.
The Squeeze Reasoning pipeline: features are squeezed into a global vector, processed via node graph reasoning, and reconstructed into a refined feature map.

Key Insight: Semantic Sparsity

A key discovery in this research is that different semantic classes (like cars vs. vegetation) naturally activate different groups of channels sparsely. This confirms that reasoning in the channel/node space is a valid and efficient proxy for reasoning in the spatial space. By treating groups of channels as "semantic nodes," the model performs reasoning at a higher level of abstraction, bypassing the expensive $O(N^2)$ pixel-affinity matrices.

Visualization of channel activations for different semantic classes, showing that specific channel indices correspond to distinct object categories.

Experimental Results

The SR module was evaluated across several diverse benchmarks, including Cityscapes, ADE20K, and MS COCO:

  • Accuracy vs. Efficiency: On Cityscapes, adding the SR head to a dilated FCN baseline improved mIoU from 74.8% to 79.9%, while only adding 1.5% to the total computational cost.
  • Real-time Performance: SRNet achieved 77.5% mIoU on Cityscapes while running at 65 FPS on a single 1080-Ti GPU, outperforming many specialized real-time models.
  • Generalization: On MS COCO, the SR module consistently outperformed Non-local blocks across object detection, instance segmentation, and panoptic segmentation tasks, all while using significantly less memory and computation.

Conclusion

Squeeze Reasoning offers a lightweight, modular framework for efficient scene understanding. By shifting the reasoning process to a compressed semantic space, it enables high-performance context modeling at a fraction of the cost of traditional methods. Its plug-and-play nature makes it easily adaptable to existing architectures for a wide range of downstream vision tasks.

Summary updated

Abstract

Graph-based convolutional model such as non-local block has shown to be effective for strengthening the context modeling ability in convolutional neural networks (CNNs). However, its pixel-wise computational overhead is prohibitive which renders it unsuitable for high resolution imagery. In this paper, we explore the efficiency of context graph reasoning and propose a novel framework called Squeeze Reasoning. Instead of propagating information on the spatial map, we first learn to squeeze the input feature into a channel-wise global vector and perform reasoning within the single vector where the computation cost can be significantly reduced. Specifically, we build the node graph in the vector where each node represents an abstract semantic concept. The refined feature within the same semantic category results to be consistent, which is thus beneficial for downstream tasks. We show that our approach can be modularized as an end-to-end trained block and can be easily plugged into existing networks. Despite its simplicity and being lightweight, the proposed strategy allows us to establish the considerable results on different semantic segmentation datasets and shows significant improvements with respect to strong baselines on various other scene understanding tasks including object detection, instance segmentation and panoptic segmentation. Code is available at https://github.com/lxtGH/SFSegNets.
BibTeX
@article{li2020towards,
  author = {Xiangtai Li and Xia Li and Ansheng You and Li Zhang and Guangliang Cheng and Kuiyuan Yang and Yunhai Tong and Zhouchen Lin},
  title = {{Towards Efficient Scene Understanding via Squeeze Reasoning}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  doi = {10.1109/tip.2021.3099369},
}
Figure