Abstract
The non-local module works as a particularly useful technique for semantic segmentation while criticized for its prohibitive computation and GPU memory occupation. In this paper, we present Asymmetric Non-local Neural Network to semantic segmentation, which has two prominent components: Asymmetric Pyramid Non-local Block (APNB) and Asymmetric Fusion Non-local Block (AFNB). APNB leverages a pyramid sampling module into the non-local block to largely reduce the computation and memory consumption without sacrificing the performance. AFNB is adapted from APNB to fuse the features of different levels under a sufficient consideration of long range dependencies and thus considerably improves the performance. Extensive experiments on semantic segmentation benchmarks demonstrate the effectiveness and efficiency of our work. In particular, we report the state-of-the-art performance of 81.3 mIoU on the Cityscapes test set. For a 256x128 input, APNB is around 6 times faster than a non-local block on GPU while 28 times smaller in GPU running memory occupation. Code is available at: https://github.com/MendelXu/ANN.git.
Introduction
Semantic segmentation—the task of assigning a category label to every pixel in an image—requires a deep understanding of both local details and global context. While standard Convolutional Neural Networks (CNNs) are great at local patterns, they struggle to capture "long-range dependencies" (e.g., understanding that a pixel in the top left belongs to a "building" because of a similar structure in the bottom right).
To fix this, researchers previously introduced Non-local (NL) blocks, which allow every pixel to "attend" to every other pixel in the image. However, NL blocks have a fatal flaw: their complexity is $O(N^2)$, where $N$ is the number of pixels. For high-resolution images, this leads to prohibitive GPU memory usage and slow processing speeds, often forcing researchers to use lower-resolution features and losing vital detail.
Methodology
The authors propose two novel components to make non-local operations "asymmetric" and efficient:
Asymmetric Pyramid Non-local Block (APNB)
Instead of comparing every pixel to every other pixel ($N \times N$), APNB samples a small, representative set of "anchor points" ($S$) using Spatial Pyramid Pooling (SPP). By pooling features at multiple scales (e.g., 1x1, 3x3, 6x6, and 8x8), the model captures global statistics without needing the full dense matrix. This reduces complexity to $O(N \times S)$, where $S \ll N$.
Asymmetric Fusion Non-local Block (AFNB)
This adapts the APNB logic to fuse features from different levels of the network (e.g., Stage 4 and Stage 5 of a ResNet). It allows high-level features to "query" sampled anchor points from lower-level features, ensuring that the fusion process considers long-range spatial relevance rather than just looking at the same pixel location.
Overall Architecture
The final architecture uses a ResNet-101 backbone, employs AFNB to merge feature stages, and applies APNB before the final classification layer.
Key Results
The model was evaluated on several major semantic segmentation benchmarks, including Cityscapes, ADE20K, and PASCAL Context.
- Performance: The model achieved a state-of-the-art 81.3% mIoU on the Cityscapes test set. It also outperformed previous leaders on ADE20K (45.24%) and PASCAL Context (52.8%).
- Efficiency: For a standard 256 × 128 input, the APNB is 6 times faster than a standard non-local block.
- Memory Savings: The GPU memory footprint of the APNB is 28 times smaller than the original non-local block, making it much more practical for real-world deployment.
- Qualitative Improvement: Visual results show that the "Full" model (using both APNB and AFNB) produces much more consistent object boundaries and fewer "fragmented" classification errors compared to baseline models.
Discussion and Conclusion
The "aha moment" here is that you don't need a dense $N \times N$ matrix to capture global context. A few well-chosen anchor points (the "pyramid") provide enough information to inform every pixel about the rest of the scene. Because the asymmetric blocks are so lightweight, they can be used on higher-resolution feature maps where standard non-local blocks would simply crash the GPU.
While the authors note some sensitivity to hyperparameters and hardware/software variance, these asymmetric blocks are general-purpose. Future directions could include applying them to other computer vision tasks like object tracking or video recognition where long-range context is equally vital.