HUSH: Holistic Panoramic 3D Scene Understanding using Spherical Harmonics

Jongsung Lee, Harin Park, Byeong-Uk Lee, Kyungdon JooCVPR 2025 · DOI

Introduction

Capturing a full 360° view of an indoor environment is straightforward with modern hardware, but extracting meaningful 3D information—such as depth or surface normals—remains a significant challenge. Standard computer vision models are typically optimized for flat, perspective images. When applied to 360° panoramas, usually represented via Equirectangular Projection (ERP), these models struggle with severe geometric distortions, particularly near the poles. Previous approaches have attempted to mitigate this by partitioning panoramas into patches or developing specialized convolutions, but these often sacrifice global context or introduce mathematical complexity.

HUSH (Holistic panoramic 3D scene Understanding using Spherical Harmonics) addresses these issues by treating 360° images as functions on a sphere. By leveraging Spherical Harmonics (SH) as adaptive geometric priors, HUSH provides a mathematically principled framework for panoramic scene understanding.

Methods

The HUSH framework integrates Spherical Harmonics—orthogonal functions commonly used in physics to represent signals on a sphere—directly into a deep learning architecture.

The HUSH architecture utilizes a Swin Transformer backbone and SH-based hierarchical attention to predict task-specific outputs like depth and surface normals.

The core components of the architecture include:

  • Scene-Adaptive SH Coefficients: Rather than using fixed SH bases, the model predicts coefficients for each scene, allowing the geometric priors to adapt to the specific layout of a room.
  • SH-based Hierarchical Attention: HUSH employs SH bases as queries within a Transformer-based attention mechanism. This forces the model to aggregate features through a "geometric lens," naturally accounting for the spherical nature of the input.
  • SH Basis Index Module: This module adaptively selects and emphasizes specific SH bases depending on the target task (e.g., depth estimation vs. normal prediction).
  • Unified Backbone: A Swin Transformer serves as the feature extractor, feeding into the SH modules which then branch into task-specific heads.

Key Insights

A primary innovation of HUSH is the use of geometry as an inductive bias. By using SH as queries, the model inherently understands the spherical topology of the data.

The SH Basis Index Maps demonstrate how the model learns to associate different spherical harmonic bases with specific geometric regions, such as floors versus walls, for different tasks.

The researchers observed that the model naturally learns a "task-specific dominance." As shown in the SH Basis Index Maps, different sets of SH bases are prioritized for different regions of the scene. For instance, certain bases might specialize in horizontal surfaces like floors, while others focus on vertical walls. This spatial coherence leads to cleaner results in complex areas compared to traditional learnable queries, which often lack explicit spatial meaning.

Results

HUSH was evaluated on major panoramic benchmarks, including Matterport3D, Stanford2D3D, and Structured3D.

  • State-of-the-Art Performance: On the Matterport3D dataset, HUSH achieved a 14.52% improvement in Root Mean Square Error (RMSE) and a 24.1% improvement in Absolute Relative Error (AbsRel) over previous leading methods like Elite360D.
  • Qualitative Quality: Visual inspections reveal that HUSH produces sharper object boundaries and more consistent planar surfaces. It avoids the "blurring" artifacts common in projection-based methods that operate on image patches.
  • Generalization: Beyond depth and normals, HUSH demonstrates strong performance in room layout estimation with minimal fine-tuning, outperforming several specialized models.

Limitations and Future Work

Despite its success, HUSH faces some challenges. Increasing the number of SH bases (the authors used 55) improves resolution but adds computational overhead to the attention mechanism. Additionally, the current model is optimized for indoor "Manhattan-world" geometries; future work could explore its efficacy in organic outdoor environments. Finally, the authors suggest that SH-based features could be integrated into emerging 3D representation techniques like Gaussian Splatting.

Summary updated

BibTeX
@inproceedings{lee2025hush,
  author = {Jongsung Lee and Harin Park and Byeong-Uk Lee and Kyungdon Joo},
  title = {{HUSH: Holistic Panoramic 3D Scene Understanding using Spherical Harmonics}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01547},
}
Figure