SphereFusion: Efficient Panorama Depth Estimation via Gated Fusion

Qingsong Yan, Qiang Wang, Kaiyong Zhao, Jie Chen, Bo Li, Xiaowen Chu, Fei Deng

Abstract

SphereFusion is an end-to-end framework designed for high-speed, high-accuracy panorama depth estimation. By integrating a novel "GateFuse" module, the system intelligently merges features from standard 2D equirectangular projections and spherical meshes. This approach effectively mitigates distortion and seam artifacts common in 360-degree imaging while maintaining real-time performance at 60 FPS.

Introduction

Estimating depth from 360° panoramas is essential for applications in autonomous driving and indoor robotics. However, traditional methods struggle with a fundamental trade-off. Equirectangular Projections (ERP) introduce significant distortion at the poles and discontinuities at the edges (seams). Conversely, spherical meshes provide geometric consistency but are often computationally prohibitive and lack fine-grained texture detail. SphereFusion addresses these challenges by processing panoramas in both domains simultaneously, leveraging the strengths of 2D CNNs and spherical geometry.

Methodology

The SphereFusion architecture is built upon a dual-stream design that extracts and fuses features from two distinct representations:

  1. Dual Encoders: A standard 2D CNN (ResNet50) serves as the image encoder to capture high-resolution texture details from ERP images. Simultaneously, a lightweight Mesh Encoder (ResNet18) processes the data on a spherical mesh to maintain geometric integrity.
  2. GateFuse Module: This core component, inspired by Gated Recurrent Units (GRUs), dynamically weights features from both streams. It learns to prioritize mesh features in high-distortion areas (like the poles) and texture features in less distorted regions (the equator).
  3. Spherical Domain Fusion: To ensure a seamless output, all features are projected into the spherical domain before final decoding. This prevents the "seam" artifacts typically found in 2D-only reconstruction methods.
  4. Inference Optimization: To achieve real-time speeds, the authors utilize an inference caching strategy. By pre-calculating the static topology of the spherical mesh, the model avoids expensive neighbor-searching operations during runtime.
The SphereFusion architecture utilizes dual encoders and a GateFuse module to combine 2D image features with spherical mesh geometry.

Experimental Results

The framework was evaluated on three major benchmarks: the synthetic 360D dataset and the real-world Matterport3D (M3D) and Stanford2D3D (S2D3D) datasets.

SphereFusion demonstrates a significant leap in efficiency, reaching an inference speed of 60 FPS (17ms) on an NVIDIA RTX 3090. This is orders of magnitude faster than competing high-accuracy models like OmniFusion. Despite this speed, SphereFusion remains highly competitive in accuracy, achieving top-tier results in Mean Relative Error (MRE) and Root Mean Square Error (RMSE).

SphereFusion achieves a superior balance of accuracy and speed (FPS) compared to existing state-of-the-art methods.

Qualitative analysis shows that the model produces remarkably clean point clouds. By decoding in the spherical domain, it maintains sharp edges on architectural features like door frames while ensuring global structural consistency without visible gaps.

Conclusion and Future Work

SphereFusion proves that mesh-based deep learning can be both accurate and exceptionally fast for panoramic tasks. While the current implementation requires additional GPU memory for mesh connectivity caching and faces scaling challenges at ultra-high resolutions (2K+), it sets a new standard for real-time 360° depth sensing. Future research will focus on optimizing memory overhead to support higher resolution inputs.

Summary updated

BibTeX
@inproceedings{yan2025spherefusion,
  author = {Qingsong Yan and Qiang Wang and Kaiyong Zhao and Jie Chen and Bo Li and Xiaowen Chu and Fei Deng},
  title = {{SphereFusion: Efficient Panorama Depth Estimation via Gated Fusion}},
  booktitle = {3DV},
  year = {2025},
  doi = {10.1109/3dv66043.2025.00084},
}
Figure