Bevanet: Bilateral Efficient Visual Attention Network for Real-Time Semantic Segmentation

Ping-Mao Huang, I-Tien Chao, Ping-Chia Huang, Jia-Wei Liao, Yung-Yu Chuang

TL;DR

This paper introduces BEVANet, a novel real-time semantic segmentation network that balances capturing broad contextual understanding with precise boundary detail. It achieves state-of-the-art performance by employing a Large Kernel Attention (LKA) mechanism with sparse kernel decomposition and a unique bilateral architecture for efficient feature fusion.

What Problem Does This Paper Address?

Real-time semantic segmentation, crucial for applications like autonomous driving, faces a fundamental challenge: efficiently capturing both the large-scale context needed for understanding scenes and the fine-grained details required for accurate object boundaries. Traditional convolutional neural networks struggle to achieve large receptive fields without becoming computationally expensive. While Vision Transformers offer long-range dependency modeling, they are also computationally demanding. Existing methods often compromise either speed or accuracy, particularly in refining object contours.

Methods

BEVANet tackles this challenge with several key innovations:

  • Large Kernel Attention (LKA) Mechanism: This is the core of BEVANet, designed to expand receptive fields efficiently.
  • Sparse Decomposed Large Separable Kernel Attentions (SDLSKA): Instead of dense large kernels, SDLSKA decomposes them into smaller convolutions and strip convolutions with dilation. This reduces computation while still capturing long-range dependencies.
  • Comprehensive Kernel Selection (CKS): This module dynamically adapts the receptive field by jointly considering channel and spatial attention. It intelligently fuses features from kernels of different sizes, going beyond decoupled approaches.
  • Bilateral Architecture: BEVANet employs a two-branch design:
  • High-level Branch: Focuses on semantic context by downsampling feature maps.
  • Low-level Branch: Extracts detailed contours and high-frequency features, maintaining higher resolution.
  • Frequent Branch Communication: These branches communicate continuously, enabling adaptive feature fusion.
  • Deep Large Kernel Pyramid Pooling Module (DLKPPM): This module enriches contextual features by synergistically combining dilated convolutions and LKA, mitigating information loss from traditional pooling operations.
  • Boundary Guided Adaptive Fusion (BGAF): This module intelligently fuses features from the high-level and low-level branches. It uses boundary information to guide the fusion process, ensuring accurate delineation of object boundaries.

Datasets

The paper evaluates BEVANet on two standard semantic segmentation benchmarks:

  • Cityscapes: A widely recognized dataset for urban scene parsing, featuring 2,975 training, 500 validation, and 1,525 testing images at a high resolution of 2048x1024, with 19 classes.
  • CamVid: A dataset of driving scenes with 960x720 resolution, containing 11 classes.

Key Insights and Ideas

  • Decomposing large kernels is key: The paper demonstrates that breaking down large convolutional kernels into sparser, dilated components (SDLSKA) is an effective way to achieve large receptive fields without prohibitive computational cost.
  • Joint spatial and channel attention is crucial: The CKS module highlights the importance of considering both spatial and channel dimensions simultaneously for adaptive feature fusion, leading to better performance than decoupled approaches.
  • Bilateral architecture with guided fusion works: The synergy between the high-level semantic branch and the low-level detail branch, guided by boundary information (BGAF), is essential for achieving both accuracy and precise contours.
  • Real-time performance without heavy pretraining: BEVANet shows strong results even without extensive ImageNet pretraining, suggesting its efficiency and suitability for resource-constrained environments.

Key Results

BEVANet achieves state-of-the-art performance for real-time semantic segmentation:

  • Cityscapes (with ImageNet pretraining): 81.0% mIoU at 33 FPS.
  • Cityscapes (without pretraining): 79.3% mIoU at 32.8 FPS, outperforming PIDNet.
  • CamVid: BEVANet-S achieves 83% mIoU at 79.4 FPS, demonstrating scalability and efficiency.
  • Qualitative results: BEVANet shows improved detection of small objects and better completeness in capturing scene elements compared to the PIDNet baseline.

Limitations and Future Work

  • Future Optimization: The authors suggest future work will focus on optimizing fusion strategies and further reducing computational overhead to develop even more lightweight large kernel attention models.
  • Scalability: While BEVANet-S shows good performance on CamVid, further exploration of its scalability to even more diverse and challenging datasets could be beneficial.

Summary updated

Abstract

Real-time semantic segmentation presents the dual challenge of designing efficient architectures that capture large receptive fields for semantic understanding while also refining detailed contours. Vision transformers model long-range dependencies effectively but incur high computational cost. To address these challenges, we introduce the Large Kernel Attention (LKA) mechanism. Our proposed Bilateral Efficient Visual Attention Network (BEVANet) expands the receptive field to capture contextual information and extracts visual and structural features using Sparse Decomposed Large Separable Kernel Attentions (SDLSKA). The Comprehensive Kernel Selection (CKS) mechanism dynamically adapts the receptive field to further enhance performance. Furthermore, the Deep Large Kernel Pyramid Pooling Module (DLKPPM) enriches contextual features by synergistically combining dilated convolutions and large kernel attention. The bilateral architecture facilitates frequent branch communication, and the Boundary Guided Attention Fusion (BGAF) module enhances boundary delineation by integrating spatial and semantic features under boundary guidance. BEVANet achieves real-time segmentation at 33 FPS, yielding 79.3% mIoU without pretraining and 81.0% mIoU on Cityscapes after ImageNet pretraining, demonstrating state-of-the-art performance.
BibTeX
@inproceedings{huang2025bevanet,
  author = {Ping-Mao Huang and I-Tien Chao and Ping-Chia Huang and Jia-Wei Liao and Yung-Yu Chuang},
  title = {{Bevanet: Bilateral Efficient Visual Attention Network for Real-Time Semantic Segmentation}},
  booktitle = {ICIP},
  year = {2025},
  url = {https://openreview.net/forum?id=djjExBDPMK},
}
Figure