Abstract
Panoramic (360°) imagery is vital for applications in virtual reality and autonomous systems, yet it presents significant challenges for semantic segmentation due to inherent geometric distortions and the high cost of manual annotation. We present DASC-SPT, a self-supervised learning framework designed to learn distortion-invariant representations. By employing a Spherical Projection Transformation (SPT) to synthesize panoramic distortions from planar data and a Deformation-aware Sampling Consistency (DASC) module to align features across views, our method achieves state-of-the-art performance on multiple panoramic benchmarks without requiring massive amounts of manually labeled data.
Introduction
The transition from planar to panoramic imaging introduces a major technical hurdle: distortion. When a spherical view is projected onto a 2D equirectangular plane, objects near the poles undergo significant stretching and warping. Standard convolutional neural networks (CNNs), optimized for the translation invariance of flat images, struggle to generalize to these warped geometries. Furthermore, the labor-intensive process of pixel-level labeling for 360° scenes limits the availability of large-scale supervised datasets.
Self-supervised learning (SSL) offers a solution by learning from unlabeled data. However, traditional SSL methods do not account for the unique spatial properties of panoramic views. DASC-SPT addresses this gap by teaching models to recognize objects specifically within the context of spherical geometry.
Methodology
The DASC-SPT framework introduces two core components to bridge the gap between planar pre-training and panoramic application.
Spherical Projection Transformation (SPT)
SPT allows the model to learn panoramic features using widely available planar datasets like ImageNet. It mathematically projects flat images onto a virtual sphere at varying latitudes and fields of view. This process generates synthetic distortions that mimic real-world panoramic sensors, enabling the model to develop "distortion-invariant" knowledge without requiring native 360° training data. The authors found that a "tangent" projection mode most accurately approximates light hitting a camera sensor, providing the most useful training signal.
Deformation-aware Sampling Consistency (DASC)
In a typical Siamese network (like SimSiam), the model enforces consistency between two different views of the same image. In the panoramic domain, these views may have drastically different distortion profiles. DASC introduces a learnable Offset Generator that predicts pixel-wise shifts and deformations between views. By "re-aligning" these features through grid sampling before computing similarity, the model learns to maintain object identity despite geometric warping.
Experimental Setup
The model was pre-trained on a subset of ImageNet-1K (approximately 34,000 images) using the SPT strategy. Evaluation was conducted on three major panoramic datasets for downstream semantic segmentation:
- SUN360E: An indoor dataset with 14 categories.
- Stanford2D3DS: A large-scale indoor dataset featuring 13 categories.
- CVPG-Pano: An outdoor panoramic dataset focused on 20 categories.
Results and Discussion
DASC-SPT demonstrates significant improvements over both supervised and existing self-supervised baselines. On the SUN360E dataset, the model achieved a mean Intersection over Union (mIoU) of 60.76%, representing a substantial increase over the supervised baseline of 51.59% and the previous state-of-the-art SSL method, PPS (58.13%).
Qualitative analysis reveals that DASC-SPT is particularly effective at maintaining the integrity of large objects (such as beds or walls) that span high-distortion regions. While baseline models often fragment such objects into pieces, DASC-SPT produces cohesive and accurate segmentation masks.
Conclusion
DASC-SPT effectively leverages planar data to solve panoramic challenges through simulated distortions and pixel-wise feature alignment. The framework proves that learning how an object deforms is just as important as learning what the object is for high-fidelity segmentation. While the method shows immense promise for CNN architectures, future work will explore scaling these geometric priors to Vision Transformers and extending the framework to other panoramic tasks such as depth estimation and object detection.
Summary updated