TL;DR
This paper introduces BEVANet, a novel real-time semantic segmentation network that balances capturing broad contextual understanding with precise boundary detail. It achieves state-of-the-art performance by employing a Large Kernel Attention (LKA) mechanism with sparse kernel decomposition and a unique bilateral architecture for efficient feature fusion.
What Problem Does This Paper Address?
Real-time semantic segmentation, crucial for applications like autonomous driving, faces a fundamental challenge: efficiently capturing both the large-scale context needed for understanding scenes and the fine-grained details required for accurate object boundaries. Traditional convolutional neural networks struggle to achieve large receptive fields without becoming computationally expensive. While Vision Transformers offer long-range dependency modeling, they are also computationally demanding. Existing methods often compromise either speed or accuracy, particularly in refining object contours.
Methods
BEVANet tackles this challenge with several key innovations:
- Large Kernel Attention (LKA) Mechanism: This is the core of BEVANet, designed to expand receptive fields efficiently.
- Sparse Decomposed Large Separable Kernel Attentions (SDLSKA): Instead of dense large kernels, SDLSKA decomposes them into smaller convolutions and strip convolutions with dilation. This reduces computation while still capturing long-range dependencies.
- Comprehensive Kernel Selection (CKS): This module dynamically adapts the receptive field by jointly considering channel and spatial attention. It intelligently fuses features from kernels of different sizes, going beyond decoupled approaches.
- Bilateral Architecture: BEVANet employs a two-branch design:
- High-level Branch: Focuses on semantic context by downsampling feature maps.
- Low-level Branch: Extracts detailed contours and high-frequency features, maintaining higher resolution.
- Frequent Branch Communication: These branches communicate continuously, enabling adaptive feature fusion.
- Deep Large Kernel Pyramid Pooling Module (DLKPPM): This module enriches contextual features by synergistically combining dilated convolutions and LKA, mitigating information loss from traditional pooling operations.
- Boundary Guided Adaptive Fusion (BGAF): This module intelligently fuses features from the high-level and low-level branches. It uses boundary information to guide the fusion process, ensuring accurate delineation of object boundaries.
Datasets
The paper evaluates BEVANet on two standard semantic segmentation benchmarks:
- Cityscapes: A widely recognized dataset for urban scene parsing, featuring 2,975 training, 500 validation, and 1,525 testing images at a high resolution of 2048x1024, with 19 classes.
- CamVid: A dataset of driving scenes with 960x720 resolution, containing 11 classes.
Key Insights and Ideas
- Decomposing large kernels is key: The paper demonstrates that breaking down large convolutional kernels into sparser, dilated components (SDLSKA) is an effective way to achieve large receptive fields without prohibitive computational cost.
- Joint spatial and channel attention is crucial: The CKS module highlights the importance of considering both spatial and channel dimensions simultaneously for adaptive feature fusion, leading to better performance than decoupled approaches.
- Bilateral architecture with guided fusion works: The synergy between the high-level semantic branch and the low-level detail branch, guided by boundary information (BGAF), is essential for achieving both accuracy and precise contours.
- Real-time performance without heavy pretraining: BEVANet shows strong results even without extensive ImageNet pretraining, suggesting its efficiency and suitability for resource-constrained environments.
Key Results
BEVANet achieves state-of-the-art performance for real-time semantic segmentation:
- Cityscapes (with ImageNet pretraining): 81.0% mIoU at 33 FPS.
- Cityscapes (without pretraining): 79.3% mIoU at 32.8 FPS, outperforming PIDNet.
- CamVid: BEVANet-S achieves 83% mIoU at 79.4 FPS, demonstrating scalability and efficiency.
- Qualitative results: BEVANet shows improved detection of small objects and better completeness in capturing scene elements compared to the PIDNet baseline.
Limitations and Future Work
- Future Optimization: The authors suggest future work will focus on optimizing fusion strategies and further reducing computational overhead to develop even more lightweight large kernel attention models.
- Scalability: While BEVANet-S shows good performance on CamVid, further exploration of its scalability to even more diverse and challenging datasets could be beneficial.
Summary updated