Tangent Plane Representations for Omnidirectional Vision

39 concepts30 papers1 proposal

Standard computer vision models are designed for the "flat" world. Convolutional neural networks (CNNs) and Transformers assume that images are rectilinear grids where spatial relationships are uniform. However, 360-degree omnidirectional data is typically stored in Equirectangular Projections (ERP)—the visual equivalent of a flattened globe. This format introduces extreme geometric distortion at the poles and non-uniform sampling, making it difficult for standard models to recognize objects or estimate depth accurately.

To bridge this gap, researchers have converged on a shared strategy: instead of designing specialized "spherical" architectures, they decompose the sphere into a collection of flat, perspective-correct patches. By projecting spherical data onto tangent planes or cubemaps, we can leverage the massive power of models pre-trained on standard perspective images.

The Distortion Problem

In an equirectangular map, a single point at the North Pole is stretched across the entire top edge of the image. As Sanchez Castillo et al. note, this geometric distortion prevents standard 2D CNNs from processing data effectively without specialized kernels. Furthermore, Dai et al. demonstrate that even basic metrics like Intersection-over-Union (IoU) become biased in ERP, as a bounding box near the equator represents a different physical area than one near the poles.

The intuition behind the tangent plane approach is to treat the sphere not as one distorted map, but as a collection of "viewports." By sampling multiple rectilinear sub-views, Yang et al. show that one can extract geometric and semantic cues using models trained entirely on standard photography.

Decomposing the Sphere

There are several ways to flatten the sphere for a neural network:

  1. Cubemap Projection (CMP): The sphere is projected onto the six faces of a cube. Kuhar et al. and Jung et al. use this to eliminate polar distortion, allowing models like YOLO or perspective-based depth estimators to function without retraining. Yu et al. further argue that cubemaps are superior for inpainting because they avoid the non-uniform pixel stretching that confuses generative kernels.
  2. Tangent Images: Rather than just six faces, the sphere is covered by many overlapping, nearly distortion-free patches. Richardt et al. and Rey-Area et al. utilize this for optical flow and high-resolution depth estimation, respectively. By processing these patches independently and blending them back together, they achieve state-of-the-art results on 360-degree data using models that have never seen a panorama.
  3. Icosahedron Subdivision: Dai et al. project data onto the faces of a subdivided icosahedron, which provides a more uniform sampling of the sphere than a simple cube.

Leveraging Generative Priors

The most recent breakthrough in this area involves using tangent projections to unlock large-scale generative models like Stable Diffusion for 360-degree tasks. Li et al. demonstrate that by projecting a spherical image into overlapping tangent planes, one can perform zero-shot super-resolution. The generative model "sees" a familiar perspective view, applies its learned priors, and the results are then projected back onto the sphere.

Similarly, for video generation, Fang et al. and Liu et al. use rotating viewports or "ViewPoint maps" to generate consistent panoramic content. By denoising individual perspective viewports and ensuring they overlap, these frameworks maintain spatial continuity across the entire 360-degree field of view.

Solving the Seam Problem

The primary challenge of decomposing a sphere into flat planes is the introduction of boundaries or "seams." If a network processes each cube face independently, the edges may not match up.

Researchers have proposed several clever workarounds:

  • Dual-Cubemaps: Lin et al. use two cubemaps, where one is rotated by 45 degrees. This ensures that the "seams" of the first cube fall in the "center" (the most undistorted part) of the second cube's faces.
  • Geometry-Aware Fusion: Guo et al. use a transformer-based mechanism to fuse features from overlapping patches, ensuring the network understands the global spherical geometry even while looking at local flat views.
  • Graph Optimization: Jung et al. use graph-based alignment to ensure that depth scales are consistent across different projected faces.

By treating the sphere as a collection of perspective views, this research direction effectively "tricks" standard AI into understanding the curved world, bypassing the need for specialized spherical datasets or architectures.

Key Foundational Work

Self-supervised Learning of Depth and Camera Motion from 360 ^\circ Videos (2018) — contributed 2 concept elements to this direction.
Pano2CAD: Room Layout From A Single Panorama Image (2016) — contributed 1 concept elements to this direction.
Automatic 3D Indoor Scene Modeling From Single Panorama (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cubemap_projection_d6f0eb24,
  title = {cubemap projection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cubemap-projection/rdr_d6f0eb24663b43e698a38d6ebae77bd0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_cubemap_projection_d6f0eb24_v1,
  title = {cubemap projection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cubemap-projection/rdr_d6f0eb24663b43e698a38d6ebae77bd0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake