Complementary Bi-directional Feature Compression for Indoor 360° Semantic Segmentation with Self-distillation

Zishuo Zheng, Chunyu Lin, Lang Nie, Kang Liao, Zhijie Shen, Yao Zhao

Introduction

Semantic segmentation in 360° panoramic images is a critical task for indoor scene understanding, yet it faces significant challenges due to the Equirectangular Projection (ERP) format. While ERP allows for a flat representation of spherical data, it introduces severe geometric distortions, particularly near the poles. Standard Convolutional Neural Networks (CNNs), designed for perspective images with uniform spatial properties, struggle to adapt to these non-linear distortions.

Recent state-of-the-art methods have attempted to mitigate this by using "horizontal-driven" approaches. These methods typically compress 2D features into a 1D horizontal representation through vertical compression. While this effectively handles vertical distortion (which is constant along lines of latitude), it introduces new limitations: unbalanced receptive fields and the loss of the "distortion distribution prior"—the geometric information inherent in how the image is stretched vertically.

Comparison between horizontal-only methods and the proposed bi-directional approach, highlighting the limitations in receptive fields and distortion handling.

Proposed Method: CBFC

The proposed framework, Complementary Bi-directional Feature Compression (CBFC), addresses these issues by processing features in two complementary directions simultaneously. The architecture is divided into three primary modules: Feature Extraction, Bi-directional Compression, and Ensemble Decoding with Self-Distillation.

1. Feature Extraction ($M_e$)

The process begins with a standard backbone (e.g., ResNet) to extract multi-scale features. To enhance spatial awareness, the authors introduce a lightweight Mix-MLP layer. This layer combines depth-wise convolutions with Multi-Layer Perceptrons (MLPs) to provide "size-free" positional encoding. This allows the network to understand the relative position of features within the panoramic view without being constrained by a fixed input resolution.

2. Bi-directional Compression ($M_c$)

This module is the core innovation of the paper. Instead of relying solely on vertical compression, the model generates two distinct 1D sequences:

  • Horizontal Representation ($S_{eqh}$): Created via vertical compression, this branch focuses on content perception by shrinking data toward the equator.
  • Vertical Representation ($S_{eqv}$): Created via horizontal compression, this branch focuses on capturing the distortion priors by contracting data toward a meridian.

To aggregate local and global context before flattening these features, a Pyramid Pooling Compression (PPC) layer is employed. This dual-path approach ensures that the model maintains a wide horizontal receptive field while simultaneously accounting for the geometric stretching of the ERP format.

The complete CBFC architecture, showing the flow from input image through the Encoder (Mix-MLP, PPC) to the Decoder (A-Conv, Self-Distillation).

3. Ensemble Decoding ($M_d$) and Self-Distillation

Converting 1D sequences back into a 2D segmentation map requires overcoming a significant "domain gap" between the horizontal and vertical features. The authors utilize "A-Conv" layers to stretch the dimensions back to 2D.

To effectively fuse these disparate features, a Self-Distillation strategy is implemented during training. The decoder consists of three branches:

  1. Horizontal-Driven Branch (HDB)
  2. Vertical-Driven Branch (VDB)
  3. Ensembled Branch (EB)

The EB acts as a "teacher," providing supervision to the HDB and VDB "students." By minimizing a combination of Cross-Entropy, KL Divergence, and L2 losses, the model forces the individual branches to learn more robust, complementary features, leading to a superior final ensemble output.

Experimental Results

The CBFC framework was evaluated on the Stanford 2D-3D-S dataset, which contains 1,413 real-world indoor panoramas. The results demonstrate a substantial performance leap over previous benchmarks:

  • State-of-the-Art Performance: At a resolution of $256 \times 512$, CBFC achieved a Mean IoU (mIoU) of 53.8%, significantly outperforming the previous leader, HoHoNet (43.3%).
  • Efficiency at Low Resolution: Even at a highly compressed resolution of $64 \times 128$, the model maintained a high mIoU, outperforming competitors by approximately 17%.
  • Qualitative Superiority: Visual inspections show that CBFC is far more effective at preserving the structural integrity of objects (like walls and ceilings) in high-distortion areas near the poles, where horizontal-only methods often fail.

Conclusion

By embracing the duality of horizontal and vertical feature compression, the CBFC framework effectively bridges the gap between content perception and geometric distortion handling in 360° images. The integration of self-distillation as a fusion mechanism further ensures that these complementary representations are maximized, setting a new standard for panoramic semantic segmentation.

Summary updated

Abstract

Semantic segmentation on 360° images is a vital component of scene understanding due to the rich surrounding information. Recently, horizontal representation-based approaches outperform projection-based solutions, because the distortions can be effectively removed by compressing the spherical data in the vertical direction. However, these methods ignore the distortion distribution prior and are limited to unbalanced receptive fields, e.g., the receptive fields are sufficient in the vertical direction and insufficient in the horizontal direction. Differently, a vertical representation compressed in another direction can offer implicit distortion prior and enlarge horizontal receptive fields. In this paper, we combine the two different representations and propose a novel 360° semantic segmentation solution from a complementary perspective. Our network comprises three modules: a feature extraction module, a bi-directional compression module, and an ensemble decoding module. First, we extract multi-scale features from a panorama. Then, a bi-directional compression module is designed to compress features into two complementary low-dimensional representations, which provide content perception and distortion prior. Furthermore, to facilitate the fusion of bi-directional features, we design a unique self distillation strategy in the ensemble decoding module to enhance the interaction of different features and further improve the performance. Experimental results show that our approach outperforms the state-of-the-art solutions on quantitative evaluations while displaying the best performance on visual appearance.
BibTeX
@inproceedings{zheng2023complementary,
  author = {Zishuo Zheng and Chunyu Lin and Lang Nie and Kang Liao and Zhijie Shen and Yao Zhao},
  title = {{Complementary Bi-directional Feature Compression for Indoor 360° Semantic Segmentation with Self-distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00448},
}
Figure