SphereCraft: A Dataset for Spherical Keypoint Detection, Matching and Camera Pose Estimation

Christiano Couto Gava, Yunmin Cho, Federico Raue, Sebastian Palacio, Alain Pagani, Andreas Dengel

Abstract

SphereCraft is a large-scale dataset designed to advance 360° (spherical) computer vision. By providing over 1.5 million image pairs with high-resolution depth maps and ground-truth correspondences, it enables researchers to train and evaluate algorithms for keypoint matching and camera pose estimation in environments where standard perspective-based models often fail.

The Challenge of Spherical Vision

While spherical images (panoramas) are becoming common in VR, robotics, and street-view mapping, most computer vision tools are still built for "perspective" images—the flat, rectangular photos captured by standard cameras.

An equirectangular projection of a 360° scene, illustrating the distortion inherent in mapping spherical data to a 2D plane.

When a 360° scene is wrapped into a flat 2D image using an equirectangular projection, it suffers from extreme distortion, especially near the poles. Existing datasets for spherical images often focus on single-view tasks like object detection or room layout prediction. However, tasks that require multiple views—like Structure from Motion (SfM) or Camera Pose Estimation—have been hindered by a lack of data providing accurate ground-truth "correspondences" (knowing exactly which pixel in Image A matches which pixel in Image B) across wide baselines and severe distortions.

The SphereCraft Dataset & Methodology

The authors developed a robust pipeline to generate both synthetic and real-world data, resulting in the SphereCraft dataset. This dataset significantly improves upon previous offerings like Matterport3D or Structured3D by providing 2048x1024 resolution, 32-bit depth maps, and explicit ground-truth keypoint correspondences.

Overview of the SphereCraft pipeline, showing 3D models, image pairs, depth maps, and ground-truth correspondences.

Data Generation Pipeline

  1. Anchor and Satellite Cameras: To simulate realistic movement, "anchor" cameras were placed in synthetic scenes with "satellite" cameras generated nearby. This creates image pairs with varying baselines, forcing models to handle significant perspective shifts.
  2. The "Shard" Strategy for Keypoints: Standard keypoint detectors (like SIFT or SuperPoint) struggle with the warping in 360° images. The authors solved this by "sharding": they tessellated the sphere into multiple overlapping flat facets (shards), ran standard detectors on these undistorted facets, and then mapped the points back onto the sphere.
  3. Ground Truth Generation: Using 32-bit float depth maps and precise camera poses from Blender, they mathematically projected points from one camera view to another. An occlusion filter was included to ensure that points hidden behind objects in one view aren't incorrectly marked as matches.

Key Insights and Benchmarking Results

The paper highlights that applying perspective algorithms directly to spherical images yields suboptimal results; the geometry of the sphere must be respected. The authors argue that scene understanding is significantly improved when reasoning about the underlying 3D structure using multiple images rather than just a single panorama.

To validate the dataset, the authors benchmarked two state-of-the-art matching algorithms: SuperGlue (designed for perspective images) and SphereGlue (a version adapted for spherical geometry).

  • SphereGlue Dominance: SphereGlue consistently outperformed SuperGlue across all metrics, including Matching Score, Precision, and Recall.
  • Pose Estimation: In camera pose estimation tasks, SphereGlue achieved significantly higher Area Under the Curve (AUC) scores. For example, in the "Shapespark" scene, SphereGlue achieved an AUC of 87.59 at a 5° threshold, while SuperGlue managed only 36.31.
  • Robustness: SphereGlue’s performance remained relatively stable across different types of scenes, whereas SuperGlue’s performance fluctuated wildly depending on the environment's complexity.

Conclusion and Future Directions

SphereCraft is a massive step forward for spherical computer vision, providing the scale and precision needed for multi-view tasks. However, the authors note several areas for future growth. Currently, the real-world portion of the dataset is used only for testing due to the lack of perfect depth maps and pose data available in synthetic sets. Future versions could include data for spherical optical flow and multi-view depth prediction. Additionally, developing a "native" spherical keypoint detector that doesn't require sharding remains an open challenge for the research community.

Summary updated

Abstract

This paper introduces SphereCraft, a dataset specifically designed for spherical keypoint detection, matching, and camera pose estimation. The dataset addresses the limitations of existing datasets by providing extracted keypoints from various detectors, along with their ground truth correspondences. Synthetic scenes with photo-realistic rendering and accurate 3D meshes are included, as well as real-world scenes acquired from different spherical cameras. SphereCraft enables the development and evaluation of algorithms targeting multiple camera viewpoints, advancing the state-of-the-art in computer vision tasks involving spherical images. Our dataset is available at https://dfki.github.io/spherecraftweb/.
BibTeX
@inproceedings{gava2024spherecraft,
  author = {Christiano Couto Gava and Yunmin Cho and Federico Raue and Sebastian Palacio and Alain Pagani and Andreas Dengel},
  title = {{SphereCraft: A Dataset for Spherical Keypoint Detection, Matching and Camera Pose Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00435},
}
Figure