PanoSwin: a Pano-style Swin Transformer for Panorama Understanding

Zhixin Ling, Zhen Xing, Xiangdong Zhou, Manliang Cao, Guichun Zhou

Abstract: In panorama understanding, the widely used equirectangular projection (ERP) entails boundary discontinuity and spatial distortion. It severely deteriorates the conventional CNNs and vision Transformers on panoramas. In this paper, we propose a simple yet effective architecture named PanoSwin to learn panorama representations with ERP. To deal with the challenges brought by equirectangular projection, we explore a pano-style shift windowing scheme and novel pitch attention to address the boundary discontinuity and the spatial distortion, respectively. Besides, based on spherical distance and Cartesian coordinates, we adapt absolute positional embeddings and relative positional biases for panoramas to enhance panoramic geometry information. Realizing that planar image understanding might share some common knowledge with panorama understanding, we devise a novel two-stage learning framework to facilitate knowledge transfer from the planar images to panoramas. We conduct experiments against the state-of-the-art on various panoramic tasks, i.e., panoramic object detection, panoramic classification, and panoramic layout estimation. The experimental results demonstrate the effectiveness of PanoSwin in panorama understanding.

Introduction

Most computer vision models are designed for "planar" images—the flat, rectangular photos typical of standard photography. However, panoramas are typically represented using Equirectangular Projection (ERP), which maps a sphere onto a flat plane. This projection introduces two major challenges for standard neural networks:

  1. Boundary Discontinuity: In a panorama, the far-left edge is adjacent to the far-right edge on the sphere, and the top/bottom edges converge at the poles. Standard models treat these as hard stops, losing the "wrap-around" context.
  2. Spatial Distortion: Objects near the poles are severely stretched compared to those at the equator. A standard square convolutional kernel or Transformer window perceives a distorted version of the world at high latitudes.

While previous research has attempted to adapt convolutions to spherical space, such methods are often computationally expensive or difficult to scale. PanoSwin addresses these issues by adapting the Swin Transformer architecture specifically for the spherical geometry of panoramas.

Methods

PanoSwin builds upon the hierarchical window-based attention of the Swin Transformer, introducing three key innovations to handle ERP-specific challenges.

Overview of PanoSwin's core components: shifted windows to handle boundary discontinuity and pitch attention via rotated panoramas to address spatial distortion.

Pano-style Shift Windowing (PSW)

Standard Swin Transformers shift windows to allow cross-window communication. PSW extends this by implementing a horizontal shift (to bridge the left-right boundary) and a "split-and-rotate" vertical shift to bridge the polar boundaries. This ensures that pixels near the "seams" of the ERP can still attend to their true neighbors on the sphere.

Comparison between the proposed PanoSwin shift windowing scheme and the original Swin scheme. PanoSwin includes horizontal and vertical rotations to maintain spherical continuity.

Pitch Attention (PA)

To combat distortion at the poles, the authors created a "plug-and-play" module called Pitch Attention. It processes the original panorama alongside a version rotated by 90 degrees (where the distorted poles are moved to the undistorted equator). By performing cross-attention between these two views, the model allows a distorted window to "learn" from its undistorted counterpart.

Pano-style Positional Encodings

Instead of standard 2D coordinates, PanoSwin uses great-circle distance for relative positional biases and Cartesian coordinates ($x, y, z$) for absolute embeddings. This embeds the actual 3D geometry of the sphere directly into the feature space.

Two-Stage Learning

Recognizing that panoramas are scarce compared to planar images, the authors use a "Planar Stage" (pre-training on augmented planar views) followed by a "Panoramic Stage" using a Knowledge Preservation (KP) loss to transfer general visual features to the spherical domain without losing accuracy in the less-distorted equatorial regions.

Experiments and Results

The model was evaluated on several panoramic benchmarks:

  • Classification: On SPH-CIFAR10, PanoSwinT+ achieved 75.01% accuracy, significantly higher than the vanilla SwinT (72.64%) and previous SOTA spherical models like SpherePHD (59.20%).
  • Object Detection: On the 360-Indoor dataset, it reached 29.4 mAP, a substantial improvement over the ResNet50 baseline (20.6 mAP) and the standard SwinT (24.0 mAP).
  • Layout Estimation: On Stanford 2D-3D, PanoSwin demonstrated its effectiveness in reconstructing room geometry from single 360° images.

Despite the extra modules, PanoSwin remains efficient. Inference speed is comparable to a standard Swin Transformer and much faster than "kernel-sampling" methods which rely on slow iterative loops over different latitudes.

Discussion and Key Insights

The primary insight of PanoSwin is that geometry matters: standard window shifting is fundamentally "geometry-blind" on a sphere. By simply changing how windows wrap around, the model captures the true topology of the image. Additionally, the "Undistortion" Trick in Pitch Attention provides a clever way to bypass complex spherical convolutions by changing the data view rather than the kernel math.

While PanoSwin excels at object detection, gains in layout estimation were more modest, likely because layout estimation relies more on long-range wall corners than polar continuity. Future work will extend PanoSwin to other panoramic tasks such as semantic segmentation and depth estimation, where pixel-level geometric accuracy is even more critical.

Summary updated

Abstract

In panorama understanding, the widely used equirectangular projection (ERP) entails boundary discontinuity and spatial distortion. It severely deteriorates the conventional CNNs and vision Transformers on panoramas. In this paper, we propose a simple yet effective architecture named PanoSwin to learn panorama representations with ERP. To deal with the challenges brought by equirectangular projection, we explore a pano-style shift windowing scheme and novel pitch attention to address the boundary discontinuity and the spatial distortion, respectively. Besides, based on spherical distance and Cartesian coordinates, we adapt absolute positional embeddings and relative positional biases for panoramas to enhance panoramic geometry information. Realizing that planar image understanding might share some common knowledge with panorama understanding, we devise a novel two-stage learning framework to facilitate knowledge transfer from the planar images to panoramas. We conduct experiments against the state-of-the-art on various panoramic tasks, i.e., panoramic object detection, panoramic classification, and panoramic layout estimation. The experimental results demonstrate the effectiveness of PanoSwin in panorama understanding.
BibTeX
@inproceedings{ling2023panoswin,
  author = {Zhixin Ling and Zhen Xing and Xiangdong Zhou and Manliang Cao and Guichun Zhou},
  title = {{PanoSwin: a Pano-style Swin Transformer for Panorama Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=JNiX0l3hFN},
}
Figure