Distortion Aware Convolutions for Omnidirectional Vision

20 concepts16 papers1 proposal

Standard convolutional neural networks (CNNs) operate under a fundamental assumption: the input data exists on a flat, Euclidean plane. On a standard pixel grid, a $3 \times 3$ convolutional kernel covers the exact same physical area whether it sits in the center of the image or near the border.

However, when we capture 360° omnidirectional images, we must project a spherical world onto a flat 2D surface. The most common format, the equirectangular projection (ERP), breaks the Euclidean assumption entirely. As noted by Cao et al. in their survey of omnidirectional representation learning, ERP introduces severe, non-uniform geometric stretching along the latitude lines.

To build an intuition for this, think of a standard world map projection. Greenland appears roughly the same size as Africa, even though Africa is physically fourteen times larger. In an ERP image, a single pixel near the equator represents a small, compact solid angle of the environment, whereas a pixel near the poles is stretched horizontally across the entire width of the image.

If a standard $3 \times 3$ convolutional kernel slides across an ERP image, its receptive field covers a tiny patch at the equator but spans a massive, distorted physical area near the poles. This destroys translation invariance: a feature (such as a window or a human face) looks entirely different to the network depending on its latitude. Consequently, standard networks struggle with tasks like object detection (Chou et al.) or layout reconstruction (Rao et al.).

To solve this, researchers developed distortion-aware and spherical convolutions. The core idea is to decouple the convolutional kernel from the flat pixel grid and define its sampling pattern based on the underlying geometry of the sphere.


Mechanism 1: Latitude-Dependent Kernel Adaptation

Because the geometric stretching in an ERP image is strictly a function of latitude, every pixel sharing the same vertical coordinate (row) undergoes the exact same amount of distortion. This predictable structure allows networks to adapt their kernels row by row.

Instead of learning entirely separate weights for every single location—which would cause an explosion in parameter count—Su et al. introduced Kernel Transformer Networks (KTNs). KTNs utilize row-dependent kernel adaptation, learning a function that dynamically transforms standard planar weights to match the specific distortion of each latitude row. This allows researchers to take "off-the-shelf" models trained on flat images and apply them directly to spherical data with minimal memory overhead.

Standard 2D Grid (Equator)          Stretched Sampling (Poles)
      [ • ][ • ][ • ]                    [  •  ][  •  ][  •  ]
      [ • ][ × ][ • ]        ───►        [  •  ][  ×  ][  •  ]
      [ • ][ • ][ • ]                    [  •  ][  •  ][  •  ]

Another way to implement this is by altering the sampling locations themselves rather than the weights. In SphereNet, developed by Coors et al., the grid locations of the convolutional filter are adjusted based on spherical coordinates. As the kernel moves closer to the poles, its sampling points stretch out horizontally. This ensures that the physical area sampled by the kernel remains constant across all latitudes, preserving feature consistency across the entire 360-degree field of view (Berenguel-Baeta et al.).


Mechanism 2: Tangent Planes and Deformable Offsets

A second intuitive approach is to imagine placing a flat tangent plane against the sphere at the target pixel's location. By projecting the kernel's regular grid onto this tangent plane and then mapping those points back to the distorted ERP pixel space, the network can sample pixels uniformly.

Yan et al. leverage this concept in 360Recon, using a variable sampling pattern on a local tangent plane at each pixel. This ensures uniform feature sampling across the entire image, which significantly improves feature matching accuracy for 3D scene reconstruction.

      [ Tangent Plane ]
            \ │ /  (Project grid points back to sphere)
             ▼▼▼
          ╭───────╮
          │ Sphere│
          ╰───────╯

This geometric projection can be combined with learnable parameters to handle both projection distortion and object shape variations. Sun et al. introduced Spherical Deformable Convolutions (SDC), which use a gnomonic projection to establish a baseline receptive field that maintains a constant physical area. They then introduce constrained, learnable offsets on top of this baseline. This allows the kernel to adapt to local semantic features while strictly compensating for the underlying spherical distortion. This builds upon earlier work by Fernandez-Labrador et al., who utilized fixed, geometry-driven offsets to provide a coherent representation for indoor layout recovery.


Mechanism 3: Direct Operations on the Unit Sphere

For complete geometric fidelity, some architectures bypass the 2D projection grid entirely during the convolution step, operating directly on a true spherical representation.

By using rotation matrices to sample points on a unit sphere, the network maintains a perfectly consistent receptive field regardless of orientation (Liu et al.). This approach eliminates not only latitude distortion but also boundary disconnectivity—the artificial seam created where the left and right edges of a panoramic image meet. Convolving directly on the sphere allows the kernel to seamlessly slide across this boundary.

However, calculating these spherical coordinates and rotation matrices for every pixel can be computationally expensive. To optimize performance, Liu et al. proposed a hybrid teacher-student framework that adaptively blends features:

  • Standard planar convolutions are utilized in the low-distortion equatorial zones, maximizing computational efficiency.
  • Spherical kernels are reserved for the extreme distortion zones found at the poles.

Why It Works: Restoring Translation Invariance

By embedding geometric awareness directly into the convolutional operator, these models achieve true distortion invariance. Standard convolutions treat panoramic images as flat planes, which causes spatial features to warp and break down near the poles. In contrast, spherical operators maintain strict spatial consistency across the entire field of view (Kato et al.).

This geometric correction yields two massive advantages:

  1. High-Latitude Reliability: Feature extraction remains robust even in the most distorted regions of the image, allowing models to accurately track human viewports (Li et al.) or estimate depth near the poles (Liu et al.).
  2. Zero-Shot Domain Transfer: Because the modified kernels neutralize distortion, models trained on standard, flat perspective image datasets (like ImageNet or COCO) can be transferred directly to omnidirectional images without requiring massive, expensive 360° retraining datasets (Coors et al.).

Key Foundational Work

Viewport Proposal CNN for 360deg Video Quality Assessment. (2019) — contributed 2 concept elements to this direction.
360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images (2019) — contributed 1 concept elements to this direction.
SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spherical_convolution_eb9db18a,
  title = {spherical convolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spherical-convolution/rdr_eb9db18afe6345b19745e56fd667ecee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_spherical_convolution_eb9db18a_v1,
  title = {spherical convolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spherical-convolution/rdr_eb9db18afe6345b19745e56fd667ecee/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake