Abstract
Semantic segmentation is one of the core tasks in the field of computer vision, and its goal is to accurately classify each pixel in an image. The traditional Unet model achieves efficient feature extraction and fusion through an encoder-decoder structure, but it still has certain limitations when dealing with complex backgrounds, long-distance dependencies, and multi-scale targets. To this end, this paper proposes an improved Unet model combined with an attention mechanism, introduces channel attention and spatial attention modules, enhances the model's ability to focus on important features, and optimizes skip connections through a multi-scale feature fusion strategy, thereby improving the combination of global semantic information and fine-grained features. The experiment is based on the Cityscapes dataset and compared with classic models such as FCN, SegNet, DeepLabv3+, and PSPNet. The improved model performs well in terms of mIoU and pixel accuracy (PA), reaching 76.5% and 95.3% respectively. The experimental results verify the superiority of this method in dealing with complex scenes and blurred target boundaries.
Introduction
Semantic segmentation—the task of assigning a category to every single pixel in an image—is foundational for autonomous driving, remote sensing, and medical imaging. While the classic Unet architecture is a staple in the field due to its elegant encoder-decoder structure, it faces significant challenges in complex urban environments. Specifically, traditional Unet models often struggle with complex backgrounds, a lack of global context for long-distance dependencies, and inconsistency when objects (like cars or pedestrians) appear at vastly different scales within the same frame. This paper addresses these "blind spots" by ensuring the network can identify both what features are important and where they are located.
Methodology
The proposed Optimized Unet enhances the standard architecture through a hybrid attention mechanism and refined feature fusion.
Architecture Overview
The model utilizes a Segformer-based encoder to extract robust features at multiple scales. As illustrated in the architecture diagram below, the network integrates CNN blocks and attention modules to process information through the encoder-decoder stages.
Hybrid Attention Mechanism
Instead of simple skip connections, which can pass redundant or noisy information, the authors insert a hybrid module that combines:
- Channel Attention Module (CAM): This uses global average pooling to calculate weights for each feature channel, identifying which semantic filters (e.g., texture vs. color) are most relevant.
- Spatial Attention Module (SAM): This applies max and average pooling across the channel dimension to create a spatial map, highlighting the most salient regions in the image.
Multi-Scale Feature Fusion and Loss
The model integrates these attention-weighted features into the skip connections, ensuring high-level semantic information from the decoder is better aligned with fine-grained spatial details from the encoder. To handle class imbalance, a weighted combination of Cross-Entropy Loss and Dice Loss was employed.
Experimental Results
The model was evaluated on the Cityscapes dataset, a high-standard benchmark for urban scene understanding containing 5,000 high-resolution images with pixel-level annotations for 19 categories.
Performance Comparison
The Optimized Unet showed superior performance across the board compared to classic and advanced models. As shown in the table below, the model achieved a mean Intersection over Union (mIoU) of 76.5% and a Pixel Accuracy (PA) of 95.3%.
Training and Convergence
Experimental results highlighted that a moderate learning rate of 0.0005 provided the optimal balance for convergence. Loss curves indicated that the model reached stable convergence after approximately 175 epochs, suggesting that the regularization strategies (Dropout and L2) were effective in preventing overfitting.
Conclusion
This research demonstrates that integrating channel and spatial attention into the Unet framework significantly improves the accuracy of multi-scale semantic segmentation. By filtering skip connections, the model effectively suppresses background noise and focuses on critical object boundaries. Future work will focus on improving inference efficiency for real-time applications and exploring the integration of Transformer-style self-attention to further enhance global context understanding.
Authors: Xuan Li, Quanchao Lu, Yankaiqi Li, Muqing Li, and Yijiashun Qi.
Summary updated