TL;DR
This paper introduces a novel approach for 360° semantic segmentation by combining horizontal and vertical feature representations. This "bi-directional compression" captures both content and distortion information, overcoming limitations of previous methods. A self-distillation strategy further enhances feature interaction, leading to state-of-the-art performance.
What Problem Does This Paper Address?
Understanding 360° indoor scenes is crucial for applications like robotics and augmented reality. Semantic segmentation, which assigns a category label to each pixel, is a key task. While horizontal-representation methods have shown promise by compressing spherical data to mitigate distortions, they suffer from unbalanced receptive fields (good vertically, poor horizontally) and neglect the implicit distortion prior. This paper aims to address these limitations by leveraging complementary horizontal and vertical representations to achieve more robust and accurate 360° semantic segmentation.
Methods
The proposed method consists of three main modules:
- Feature Extraction Module: A standard deep convolutional neural network (like ResNet) extracts multi-scale hierarchical features from the input equirectangular projection (ERP) 360° image.
- Bi-directional Compression Module: This novel module compresses the extracted features into two complementary low-dimensional representations:
- Horizontal Representation: Achieved through a Pyramid Pooling Compression (PPC) layer that aggregates sub-regions with varying receptive fields, capturing global context and distortion information.
- Vertical Representation: Also uses a PPC layer, but focuses on capturing implicit distortion prior and enhancing horizontal receptive fields.
- A Mix-MLP Layer is used before compression to inject implicit location information, addressing the limitations of fixed positional encodings.
- Ensemble Decoding Module: This module reconstructs 2D feature maps from the compressed representations. Crucially, it employs a unique self-distillation strategy to fuse the horizontal and vertical features. This strategy treats the individual horizontal and vertical branches as "student" models and their fused output as the "teacher" model. This allows students to learn from the teacher and vice-versa, fostering interaction and narrowing the feature domain gap between the two representations. The distillation process uses cross-entropy, KL divergence, and L2 loss.
Datasets
The primary dataset used for training and evaluation is the Stanford 2D-3D-S dataset. This dataset comprises 1413 real-world equirectangular RGB-D images with 13 indoor scene categories. The paper reports results using 3-fold cross-validation.
Key Insights and Ideas
- Complementarity is Key: The core insight is that horizontal and vertical representations capture different, complementary information about 360° scenes. Combining them is more effective than relying on a single direction.
- Implicit Distortion Prior: The vertical representation, by compressing along a different axis, implicitly captures distortion distribution information that is missed by purely horizontal methods.
- Self-Distillation for Fusion: Instead of complex fusion networks, self-distillation is an elegant way to bridge the domain gap between the two representations and improve their interaction. This is a novel application of self-distillation for feature fusion in this context.
- Location Matters: Incorporating location information via the Mix-MLP layer is crucial for effectively processing ERP images.
Key Results
- Quantitative Performance: The proposed method significantly outperforms state-of-the-art approaches on the Stanford 2D-3D-S dataset. For instance, at the lowest resolution (64x128), it achieves a 17% improvement in mIoU and mAcc compared to a horizontal-only representation method. On higher resolutions, it also shows substantial gains, achieving 56.7% mIoU and 70.8% mAcc on RGB-D input at 512x1024.
- Qualitative Performance: Visual results demonstrate that the proposed method handles distortions and complex contextual information better than previous methods, producing more accurate and complete segmentation maps.
Limitations and Future Work
- Device Limitations: The authors mention that due to device limitations, they were unable to train their network on higher resolutions as some prior works did. However, they show competitive performance even on lower resolutions with a stronger backbone (ResNet-101).
- Future Work: While not explicitly detailed, the success of the bi-directional approach and self-distillation suggests that further exploration of complementary representations and advanced fusion techniques could lead to even greater improvements. The paper's focus on indoor scenes might also inspire extensions to outdoor or mixed environments.
Summary updated