Introduction
Semantic segmentation in 360° panoramic images is a critical task for indoor scene understanding, yet it faces significant challenges due to the Equirectangular Projection (ERP) format. While ERP allows for a flat representation of spherical data, it introduces severe geometric distortions, particularly near the poles. Standard Convolutional Neural Networks (CNNs), designed for perspective images with uniform spatial properties, struggle to adapt to these non-linear distortions.
Recent state-of-the-art methods have attempted to mitigate this by using "horizontal-driven" approaches. These methods typically compress 2D features into a 1D horizontal representation through vertical compression. While this effectively handles vertical distortion (which is constant along lines of latitude), it introduces new limitations: unbalanced receptive fields and the loss of the "distortion distribution prior"—the geometric information inherent in how the image is stretched vertically.
Proposed Method: CBFC
The proposed framework, Complementary Bi-directional Feature Compression (CBFC), addresses these issues by processing features in two complementary directions simultaneously. The architecture is divided into three primary modules: Feature Extraction, Bi-directional Compression, and Ensemble Decoding with Self-Distillation.
1. Feature Extraction ($M_e$)
The process begins with a standard backbone (e.g., ResNet) to extract multi-scale features. To enhance spatial awareness, the authors introduce a lightweight Mix-MLP layer. This layer combines depth-wise convolutions with Multi-Layer Perceptrons (MLPs) to provide "size-free" positional encoding. This allows the network to understand the relative position of features within the panoramic view without being constrained by a fixed input resolution.
2. Bi-directional Compression ($M_c$)
This module is the core innovation of the paper. Instead of relying solely on vertical compression, the model generates two distinct 1D sequences:
- Horizontal Representation ($S_{eqh}$): Created via vertical compression, this branch focuses on content perception by shrinking data toward the equator.
- Vertical Representation ($S_{eqv}$): Created via horizontal compression, this branch focuses on capturing the distortion priors by contracting data toward a meridian.
To aggregate local and global context before flattening these features, a Pyramid Pooling Compression (PPC) layer is employed. This dual-path approach ensures that the model maintains a wide horizontal receptive field while simultaneously accounting for the geometric stretching of the ERP format.
3. Ensemble Decoding ($M_d$) and Self-Distillation
Converting 1D sequences back into a 2D segmentation map requires overcoming a significant "domain gap" between the horizontal and vertical features. The authors utilize "A-Conv" layers to stretch the dimensions back to 2D.
To effectively fuse these disparate features, a Self-Distillation strategy is implemented during training. The decoder consists of three branches:
- Horizontal-Driven Branch (HDB)
- Vertical-Driven Branch (VDB)
- Ensembled Branch (EB)
The EB acts as a "teacher," providing supervision to the HDB and VDB "students." By minimizing a combination of Cross-Entropy, KL Divergence, and L2 losses, the model forces the individual branches to learn more robust, complementary features, leading to a superior final ensemble output.
Experimental Results
The CBFC framework was evaluated on the Stanford 2D-3D-S dataset, which contains 1,413 real-world indoor panoramas. The results demonstrate a substantial performance leap over previous benchmarks:
- State-of-the-Art Performance: At a resolution of $256 \times 512$, CBFC achieved a Mean IoU (mIoU) of 53.8%, significantly outperforming the previous leader, HoHoNet (43.3%).
- Efficiency at Low Resolution: Even at a highly compressed resolution of $64 \times 128$, the model maintained a high mIoU, outperforming competitors by approximately 17%.
- Qualitative Superiority: Visual inspections show that CBFC is far more effective at preserving the structural integrity of objects (like walls and ceilings) in high-distortion areas near the poles, where horizontal-only methods often fail.
Conclusion
By embracing the duality of horizontal and vertical feature compression, the CBFC framework effectively bridges the gap between content perception and geometric distortion handling in 360° images. The integration of self-distillation as a fusion mechanism further ensures that these complementary representations are maximized, setting a new standard for panoramic semantic segmentation.
Summary updated