Complementary Bi-directional Feature Compression for Indoor 360° Semantic Segmentation with Self-distillation

Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Zhijie Shen, Yao Zhao

TL;DR

This paper introduces a novel approach for 360° semantic segmentation by combining horizontal and vertical feature representations. This "bi-directional compression" captures both content and distortion information, overcoming limitations of previous methods. A self-distillation strategy further enhances feature interaction, leading to state-of-the-art performance.

What Problem Does This Paper Address?

Understanding 360° indoor scenes is crucial for applications like robotics and augmented reality. Semantic segmentation, which assigns a category label to each pixel, is a key task. While horizontal-representation methods have shown promise by compressing spherical data to mitigate distortions, they suffer from unbalanced receptive fields (good vertically, poor horizontally) and neglect the implicit distortion prior. This paper aims to address these limitations by leveraging complementary horizontal and vertical representations to achieve more robust and accurate 360° semantic segmentation.

Methods

The proposed method consists of three main modules:

  1. Feature Extraction Module: A standard deep convolutional neural network (like ResNet) extracts multi-scale hierarchical features from the input equirectangular projection (ERP) 360° image.
  2. Bi-directional Compression Module: This novel module compresses the extracted features into two complementary low-dimensional representations:
  3. Horizontal Representation: Achieved through a Pyramid Pooling Compression (PPC) layer that aggregates sub-regions with varying receptive fields, capturing global context and distortion information.
  4. Vertical Representation: Also uses a PPC layer, but focuses on capturing implicit distortion prior and enhancing horizontal receptive fields.
  5. A Mix-MLP Layer is used before compression to inject implicit location information, addressing the limitations of fixed positional encodings.
  6. Ensemble Decoding Module: This module reconstructs 2D feature maps from the compressed representations. Crucially, it employs a unique self-distillation strategy to fuse the horizontal and vertical features. This strategy treats the individual horizontal and vertical branches as "student" models and their fused output as the "teacher" model. This allows students to learn from the teacher and vice-versa, fostering interaction and narrowing the feature domain gap between the two representations. The distillation process uses cross-entropy, KL divergence, and L2 loss.

Datasets

The primary dataset used for training and evaluation is the Stanford 2D-3D-S dataset. This dataset comprises 1413 real-world equirectangular RGB-D images with 13 indoor scene categories. The paper reports results using 3-fold cross-validation.

Key Insights and Ideas

  • Complementarity is Key: The core insight is that horizontal and vertical representations capture different, complementary information about 360° scenes. Combining them is more effective than relying on a single direction.
  • Implicit Distortion Prior: The vertical representation, by compressing along a different axis, implicitly captures distortion distribution information that is missed by purely horizontal methods.
  • Self-Distillation for Fusion: Instead of complex fusion networks, self-distillation is an elegant way to bridge the domain gap between the two representations and improve their interaction. This is a novel application of self-distillation for feature fusion in this context.
  • Location Matters: Incorporating location information via the Mix-MLP layer is crucial for effectively processing ERP images.

Key Results

  • Quantitative Performance: The proposed method significantly outperforms state-of-the-art approaches on the Stanford 2D-3D-S dataset. For instance, at the lowest resolution (64x128), it achieves a 17% improvement in mIoU and mAcc compared to a horizontal-only representation method. On higher resolutions, it also shows substantial gains, achieving 56.7% mIoU and 70.8% mAcc on RGB-D input at 512x1024.
  • Qualitative Performance: Visual results demonstrate that the proposed method handles distortions and complex contextual information better than previous methods, producing more accurate and complete segmentation maps.

Limitations and Future Work

  • Device Limitations: The authors mention that due to device limitations, they were unable to train their network on higher resolutions as some prior works did. However, they show competitive performance even on lower resolutions with a stronger backbone (ResNet-101).
  • Future Work: While not explicitly detailed, the success of the bi-directional approach and self-distillation suggests that further exploration of complementary representations and advanced fusion techniques could lead to even greater improvements. The paper's focus on indoor scenes might also inspire extensions to outdoor or mixed environments.

Summary updated

Abstract

Semantic segmentation on 360° images is a vital component of scene understanding due to the rich surrounding information. Recently, horizontal representation-based approaches outperform projection-based solutions, because the distortions can be effectively removed by compressing the spherical data in the vertical direction. However, these methods ignore the distortion distribution prior and are limited to unbalanced receptive fields, e.g., the receptive fields are sufficient in the vertical direction and insufficient in the horizontal direction. Differently, a vertical representation compressed in another direction can offer implicit distortion prior and enlarge horizontal receptive fields. In this paper, we combine the two different representations and propose a novel 360° semantic segmentation solution from a complementary perspective. Our network comprises three modules: a feature extraction module, a bi-directional compression module, and an ensemble decoding module. First, we extract multi-scale features from a panorama. Then, a bi-directional compression module is designed to compress features into two complementary low-dimensional representations, which provide content perception and distortion prior. Furthermore, to facilitate the fusion of bi-directional features, we design a unique self distillation strategy in the ensemble decoding module to enhance the interaction of different features and further improve the performance. Experimental results show that our approach outperforms the state-of-the-art solutions on quantitative evaluations while displaying the best performance on visual appearance.
BibTeX
@inproceedings{zheng2023complementary,
  author = {Zishuo Zheng and Chunyu Lin and Lang Nie and Kang Liao and Zhijie Shen and Yao Zhao},
  title = {{Complementary Bi-directional Feature Compression for Indoor 360° Semantic Segmentation with Self-distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00448},
}
Figure