Semantic Segmentation of Degraded Images Using Layer-Wise Feature Adjustor

Kazuki Endo, Masayuki Tanaka, Masatoshi OkutomiWACV 2023

TL;DR

This paper introduces a novel semantic segmentation network designed to handle images with varying levels of degradation (like JPEG compression, blur, or noise) without sacrificing performance on clean images. Their "layer-wise feature adjustor" leverages knowledge distillation to transfer knowledge from a clean-image-trained network to a target network, ensuring consistency across multiple feature layers.

What Problem Does This Paper Address?

Semantic segmentation, a crucial task for applications like autonomous driving, typically performs best on clean, high-quality images. However, real-world images are often degraded due to various factors such as compression (e.g., JPEG), blurring, or noise. The degree of this degradation can vary significantly and is often unknown in practical scenarios. Existing methods struggle to maintain high performance across a wide range of degradation levels while also preserving accuracy on clean images. A naive approach of training on both clean and degraded images can lead to a performance drop on clean images. This paper aims to bridge this gap by developing a segmentation network that is robust to various degradation levels without compromising its ability to segment clean images.

Methods

The core of the proposed method is a "layer-wise feature adjustor" that employs knowledge distillation. They train a "source network" solely on clean images. A "target network" is then trained to mimic the behavior of the source network. The key innovation lies in how this mimicry is achieved:

  • Layer-wise Consistency: Instead of just matching the final output features, the target network is trained to maintain consistency with the source network's features across multiple layers of its feature extractor.
  • Feature Adjustor: For each layer in the target network's feature extractor, a trainable "feature adjustor" is introduced. This adjustor takes the features from the target network's current layer and learns to scale and bias them to match the corresponding features from the source network. This is achieved using scale and bias estimators.
  • Mixed Training: The target network is trained using a "mixed training" strategy, where it receives both clean and degraded images with varying degradation levels.
  • Knowledge Distillation: The overall training objective combines a consistency loss (based on cosine similarity) across multiple layers with the standard segmentation loss.

The authors use SegNet as the base architecture for both the source and target networks.

Datasets

The proposed method was evaluated on two standard datasets:

  • CamVid: A road scene dataset commonly used for autonomous driving applications. The authors used static images from 11 categories.
  • SUN RGB-D: An indoor scene dataset that includes RGB-D images. The authors utilized only the RGB images from 37 categories.

The paper does not introduce new datasets but rather demonstrates the effectiveness of their method on existing benchmarks.

Key Insights and Ideas

  • Multi-layer Consistency is Key: Matching features across multiple layers, not just the final layer, significantly improves robustness to degradation while preserving clean image performance. This suggests that degradation affects features at various levels of abstraction.
  • Degradation-Aware Feature Adjustment: The learnable scale and bias adjustors are crucial for adapting the target network's features to match the clean source features, effectively "undoing" the degradation's impact at each layer.
  • "Degrade" Network's Limitation: Training a network directly on both clean and degraded images (the "Degrade" baseline) shows a trade-off: it improves performance on degraded images but degrades performance on clean images, highlighting the need for a more sophisticated approach.
  • Effectiveness Across Degradation Types: The proposed method demonstrates robustness not only to JPEG distortion but also to Gaussian blur and salt-and-pepper noise, suggesting its general applicability.

Key Results

  • Superior Performance: The proposed "Proposed" network consistently outperformed the "Clean" (source network) and "Degrade" (mixed training) baselines across various degradation levels and datasets.
  • Clean Image Performance Preservation: For clean images, the "Proposed" network achieved comparable performance to the "Clean" network, demonstrating that it does not sacrifice accuracy on un-degraded data.
  • JPEG Distortion: On CamVid, the average mIoU for the proposed method was 0.566, outperforming "Clean" (0.547) and "Degrade" (0.534). Similar improvements were observed on SUN RGB-D.
  • Other Degradations: For Gaussian blur, the proposed method achieved an average mIoU of 0.543, significantly better than "Degrade" (0.465). For salt-and-pepper noise, it achieved 0.564, also outperforming "Degrade" (0.504).
  • Comparison with Single-Layer Adjustor: The proposed layer-wise feature adjustor outperformed a single-layer feature adjustor, especially on low-quality degraded images, showing an average mIoU improvement of 0.009 for JPEG distortion and even larger gains for blur and noise.

Limitations and Future Work

  • Computational Cost: The proposed network is larger than the source network, and the authors acknowledge the need to reduce the number of parameters.
  • Single Degradation Type: The current work focuses on handling a single type of degradation at a time.
  • Future Directions:
    • Applying the method to other tasks like object detection.
  • Reducing the model's parameter count.
  • Extending the framework to handle mixtures of degradations, which is more realistic.

Summary updated

BibTeX
@inproceedings{endo2023semantic,
  author = {Kazuki Endo and Masayuki Tanaka and Masatoshi Okutomi},
  title = {{Semantic Segmentation of Degraded Images Using Layer-Wise Feature Adjustor}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure