Inter-Level Feature Balanced Fusion Network for Street Scene Segmentation

Dongqian Li, Cien Fan, Lian Zou, Qi Zuo, Hao Jiang, Yifeng Liu2021 · DOI

TL;DR

This paper introduces the Inter-Level Feature Balanced Fusion Network (IFBFNet) for street scene semantic segmentation. It addresses the issue of unbalanced feature fusion in existing encoder-decoder architectures by proposing a novel "feature balanced fusion module" that adaptively weights low-level spatial and high-level semantic features, leading to improved boundary prediction and overall accuracy.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a category label to every pixel in an image, is crucial for applications like autonomous driving and robotics. Many state-of-the-art methods employ an encoder-decoder architecture. The encoder progressively downsamples the image to extract rich semantic information, while the decoder upsamples this information to reconstruct a pixel-level segmentation map. A common challenge is the loss of fine-grained spatial details during the encoder's downsampling process. To mitigate this, decoders often fuse low-level features (rich in spatial detail) with high-level features (rich in semantics). However, traditional fusion methods like simple concatenation or element-wise addition often lead to unbalanced feature integration, where one type of feature dominates, or information is lost, resulting in imprecise boundaries and inaccurate segmentation, especially for small objects.

Methods

The proposed IFBFNet builds upon the encoder-decoder framework.

  • Encoder: Utilizes a ResNet101 backbone with atrous convolutions to extract rich semantic features while maintaining a reasonable feature resolution. An Atrous Spatial Pyramid Pooling (ASPP) module is employed to capture multi-scale contextual information.
  • Decoder: Employs skip-connections to fuse low-level and high-level features. The core innovation is the Inter-Level Feature Balanced Fusion Module. This module takes concatenated low-level and high-level features, calculates their correlation in the channel dimension, and uses sigmoid normalization to derive adaptive weights for each feature level. These weights are then used to fuse the features, allowing the network to learn how to balance their contributions. A residual connection is also added to preserve primitive information.
  • Spatial Stream: A shallow, parallel branch with three convolutional layers is introduced to explicitly capture and supplement fine-grained spatial details that might be lost in the main encoder-decoder path. This stream's output is fused with the main semantic features before the final prediction.
  • Loss Function: The paper uses Online Hard Example Mining (OHEM) loss to focus on difficult-to-classify samples. An auxiliary loss function is also incorporated to supervise a coarse output branch, aiming to improve network optimization.

Datasets

The primary dataset used for training and evaluation is Cityscapes. This dataset contains finely annotated urban scene understanding data, with 19 classes for pixel-level segmentation. The authors specifically mention using the finely annotated set for training and validation, and also performed experiments on the test set.

Key Insights and Ideas

  • Feature Imbalance is a Bottleneck: The paper convincingly argues that simple concatenation or addition of low-level and high-level features is suboptimal because these features have fundamentally different characteristics, and naive fusion doesn't leverage them effectively.
  • Adaptive Weighting for Balanced Fusion: The "feature balanced fusion module" is the key novelty. By learning adaptive weights, it allows the network to dynamically decide how much importance to give to spatial versus semantic information at each fusion point, leading to a more nuanced and effective integration.
  • Complementary Spatial Stream: The addition of a dedicated shallow spatial stream proves beneficial, demonstrating that explicitly preserving and enhancing fine spatial details can significantly improve boundary prediction and small object segmentation.
  • Synergy of Components: The experiments show that the proposed balanced fusion module and the spatial stream, when combined with standard techniques like ASPP and OHEM, lead to substantial improvements over strong baselines.

Key Results

  • The proposed IFBFNet achieved a competitive mean Intersection over Union (mIoU) of 81.2% on the Cityscapes validation set, outperforming the baseline DeepLabV3+ by nearly 1.3%.
  • Ablation studies demonstrated the effectiveness of individual components:
  • Adding ASPP improved performance over a basic ResNet.
  • The Inter-Level Feature Balanced Fusion module improved performance by 2.06% (ResNet50) and 2.40% (ResNet101) compared to simple skip-connection concatenation.
  • The spatial stream further boosted performance.
  • Qualitative results (Figure 6) show that IFBFNet significantly improves object boundary prediction and the segmentation of small objects like power poles.
  • On the Cityscapes test set, the network achieved 79.6% mIoU, outperforming existing approaches and DeepLabV3+ by 1.1%.

Limitations and Future Work

  • Future Work: The authors suggest exploring more application scenarios for the inter-level feature balanced module, as it can be extended to integrate information from more than two levels or in other domains requiring spatial and semantic understanding. They also note that while the current design is effective for two-level fusion, extending it to three or more levels would require adding corresponding input branches and weight blocks, which needs further verification.
  • Limitations: The paper focuses on street scene segmentation. While the core module is generalizable, its performance on vastly different domains would require further testing. The paper also mentions that the auxiliary loss weight was tuned to 0.4 for optimal performance, implying that hyperparameter tuning is crucial.

Summary updated

BibTeX
@misc{li2021inter,
  author = {Dongqian Li and Cien Fan and Lian Zou and Qi Zuo and Hao Jiang and Yifeng Liu},
  title = {{Inter-Level Feature Balanced Fusion Network for Street Scene Segmentation}},
  year = {2021},
  doi = {10.3390/s21237844},
}
Figure