Complementary Bi-Directional Feature Compression for Indoor 360deg Semantic Segmentation With Self-Distillation

Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Zhijie Shen, Yao ZhaoWACV 2023

TL;DR

This paper introduces a novel approach for 360° semantic segmentation by leveraging complementary horizontal and vertical feature representations. By compressing panoramic images in both directions and using a self-distillation strategy to fuse these representations, the model achieves state-of-the-art performance, outperforming existing methods in both quantitative and qualitative evaluations.

What Problem Does This Paper Address?

Understanding indoor scenes from 360° panoramic images is crucial for various applications like robotics and augmented reality. However, traditional computer vision methods struggle with the inherent distortions present in these images, especially when using equirectangular projection (ERP). While some methods focus on horizontal representations to mitigate distortions, they often suffer from limited receptive fields in the horizontal direction. Conversely, vertical representations can capture distortion distribution but might blur content. This paper addresses the challenge of effectively utilizing both types of information to achieve accurate and robust 360° semantic segmentation.

Methods

The proposed network comprises three main modules:

  1. Feature Extraction Module: This module uses a standard backbone (like ResNet) to extract multi-scale features from the input ERP image.
  2. Bi-Directional Compression Module: This is the core innovation. It compresses the extracted features into two complementary low-dimensional representations:
  3. Mix-MLP Layer: A novel layer that mixes 3x3 convolutions with zero padding and MLPs to inject implicit location information, addressing the limitations of fixed positional encodings.
  4. Pyramid Pooling Compression (PPC) Layer: This layer aggregates features from different sub-regions at various scales to capture both local and global context, effectively compressing the feature maps along both horizontal and vertical directions. This results in two distinct 1D sequences: Seq_h (horizontal representation) and Seq_v (vertical representation).
  5. Ensemble Decoding Module: This module reconstructs 2D dense feature maps from the compressed 1D representations. It uses A-Conv layers to progressively stretch dimensions. Crucially, it incorporates a self-distillation strategy to fuse the Seq_h and Seq_v representations. In this strategy, the horizontal-driven branch (HDB) and vertical-driven branch (VDB) act as "student" models, while their ensemble fusion (EB) acts as the "teacher." This allows knowledge transfer between the representations, enhancing their interaction and improving segmentation accuracy. The loss function includes cross-entropy, KL divergence, and L2 loss to facilitate this distillation.

Datasets

The primary dataset used for training and evaluation is the Stanford 2D-3D-S dataset. This dataset consists of 1413 real-world equirectangular RGB-D images with semantic labels for 13 categories. The paper reports results using 3-fold cross-validation.

Key Insights and Ideas

  • Complementary Representations are Key: The paper strongly advocates for the combined use of horizontal and vertical representations. It highlights that while horizontal representations are good at removing distortions, they lack horizontal receptive field coverage. Vertical representations, on the other hand, can capture distortion distribution and improve horizontal receptive fields, even if they slightly blur content. Combining them offers a more holistic understanding.
  • Implicit Distortion Prior: The vertical representation is shown to provide an "implicit distortion prior," which is valuable for accurate segmentation in distorted panoramic images.
  • Self-Distillation for Fusion: Directly fusing disparate feature domains (horizontal and vertical) is challenging. The proposed self-distillation method effectively bridges this gap by allowing student branches (HDB, VDB) to learn from the fused teacher branch (EB), leading to better feature interaction and improved performance without significantly increasing model complexity.
  • Mix-MLP for Location Awareness: The novel Mix-MLP layer is crucial for injecting location information into the features, which is vital for understanding the unique structure of ERP images, especially when dealing with 1D compressed representations.

Key Results

  • State-of-the-Art Performance: The proposed method significantly outperforms existing state-of-the-art methods on the Stanford 2D-3D-S dataset across various resolutions. For example, on the lowest resolution (64x128), it achieves an mIoU of 47.2 and mAcc of 61.2, outperforming previous methods by a considerable margin (e.g., 17% improvement on mIoU and mAcc over HoHoNet [33]).
  • Improved Accuracy with Resolution: The model shows consistent improvements with increasing resolution, achieving 53.8 mIoU and 66.5 mAcc on 1024x2048 resolution.
  • Efficiency: Despite its advanced capabilities, the model maintains acceptable computational complexity, offering a good trade-off between performance and efficiency compared to other methods.
  • Qualitative Improvements: Visual results demonstrate that the proposed method handles distortions and complex contextual information better than purely horizontal representation-based methods, producing more accurate segmentation maps, especially in challenging regions.

Limitations and Future Work

  • Resolution Limitations: The authors mention that due to hardware limitations, they were unable to train their network on the highest resolutions (2048x4096) as some prior works did. They hypothesize that at very high resolutions, the 1D compression might make recovering 2D information more challenging.
  • Future Work: The paper suggests that the proposed complementary representation and self-distillation strategy can be extended to other 360° vision tasks, such as depth estimation and scene layout prediction. The authors also imply that further exploration of different pooling strategies within the PPC layer or alternative compression techniques could yield further improvements.

Summary updated

BibTeX
@inproceedings{zheng2023complementary,
  author = {Zishuo Zheng and Chunyu Lin and Lang Nie and Kang Liao and Zhijie Shen and Yao Zhao},
  title = {{Complementary Bi-Directional Feature Compression for Indoor 360deg Semantic Segmentation With Self-Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure