Standard computer vision models are fundamentally flat-earthers. They assume the visual world is structured as a uniform Euclidean grid where a $3 \times 3$ pixel window means the exact same thing whether it sits at the center or the edge of an image. This property—known as translation invariance—is the bedrock of standard convolutional neural networks (CNNs) and vision transformers (ViTs).
However, when we capture a full $360^\circ$ environment, we must flatten a sphere onto a 2D plane. The most common format for storing these panoramas is the Equirectangular Projection (ERP). Much like the Mercator projection of a world map, ERP introduces severe, non-uniform geometric distortion. Near the equator, objects retain their shape. As you move toward the poles, pixels stretch horizontally until a single point at the north pole spans the entire width of the image.
As noted by Haozheng Yu et al. in PanelNet: Understanding 360 Indoor Environment via Panel Representation, this stretching completely breaks the spatial assumptions of standard networks. Because an object's scale and shape change drastically depending on its vertical coordinate (latitude), fixed-weight kernels cannot extract consistent features across the image. Over the past few years, researchers have developed several distinct strategies to restore spatial sanity to spherical imagery.
Strategy 1: Deforming the Convolutional Kernel
If the pixels in an image are stretched non-uniformly, the most direct solution is to stretch the convolutional filters to match. Instead of sampling from a rigid square grid, spherical convolutions dynamically adjust their sampling locations based on the latitude of the target pixel.
Standard Convolution Spherical Convolution (At High Latitude)
[·][·][·] [·] [·] [·]
[·][x][·] [·][x][·]
[·][·][·] [·] [·] [·]
(Uniform grid spacing) (Horizontal sampling expands)
Benjamin Coors et al. pioneered this approach with SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images, projecting the kernel's receptive field directly onto a spherical surface before mapping it back to the distorted ERP grid. This ensures that the physical area covered by a kernel remains constant regardless of latitude.
Rather than engineering new architectures from scratch, Yu-Chuan Su et al. introduced Kernel Transformer Networks for Compact Spherical Convolution, which learns a function that takes standard perspective weights and continuously morphs them to match local ERP distortions. To handle these scale variations dynamically without explicit geometric remapping, Chuanqing Zhuang et al. developed ACDNet: Adaptively Combined Dilated Convolution for Monocular Panorama Depth Estimation, which uses a channel-wise attention mechanism to combine multiple dilation rates, allowing the network to adaptively expand its horizontal reach near the poles.
Strategy 2: Subdividing the Sphere into Perspective Patches
Instead of modifying the network's internal mathematical operations, an alternative approach changes how the image data is presented to the model. If neural networks excel at processing perspective images, we can decompose the sphere into a collection of flat, overlapping perspective viewports where distortion is negligible.
One classic alternative to ERP is the cubemap projection, which maps the sphere onto the six faces of a cube. While cubemaps eliminate polar stretching, they introduce sharp artificial boundaries and cause structural features to break across edges. To balance the benefits of both representations, Fu-En Wang et al. designed BiFuse: Monocular 360 Depth Estimation via Bi-Projection Fusion, a dual-branch network that shares information between a wide-angle ERP branch and a distortion-free cubemap branch. To recover the structural details often lost at cubemap boundaries, Chunyu Lin et al. proposed a dual-cubemap framework in Distortion-Tolerant Monocular Depth Estimation on Omnidirectional Images Using Dual-Cubemap, which processes two separate cubemaps rotated $45^\circ$ relative to one another to ensure seamless coverage.
[ ERP Image ] ─── (Severe Polar Distortion) ───┐
▼
[ Fusion Module ] ──► Final Output
▲
[ Cubemap ] ─── (Discontinuous Edges) ───┘
For ultra-high-resolution tasks, projecting onto a rigid cube is still too restrictive. Manuel Rey-Area et al. resolved this in 360MonoDepth: High-Resolution 360° Monocular Depth Estimation by breaking the sphere down into dozens of small, overlapping tangent-plane images. This allows off-the-shelf perspective models to perform high-fidelity depth estimation before the patches are stitched back together using global alignment algorithms. This tangent-plane strategy has also unlocked zero-shot capabilities for generative models; for example, Runyi Li et al. utilized it in OmniSSR: Zero-Shot Omnidirectional Image Super-Resolution Using Stable Diffusion Model to upscale panoramic images using pre-trained diffusion models without introducing any polar artifacts.
Strategy 3: Exploiting Structural Invariants
In man-made environments, we can exploit a powerful physical invariant: gravity alignment. When a $360^\circ$ camera is leveled, the vertical lines of a room (such as wall corners, doors, and columns) remain perfectly straight and parallel in an ERP image. Only the horizontal lines (ceilings and floors) curve.
┌────────────────────────────────────────┐ ▲ Polar Stretching
│ ) Ceiling Curve (Distorted) ( │ │ (Severe)
│ │ │ │
│ │ Vertical Wall Line (Straight)│ │ ▼ Equator
│ │ │ │ (Minimal)
└────────────────────────────────────────┘
By recognizing that the vertical axis does not suffer from structural warping, researchers can simplify 2D spatial reasoning into a series of 1D operations. Cheng Sun demonstrated this elegant simplification in HoHoNet: 360 Indoor Holistic Understanding With Latent Horizontal Features, compressing 2D feature maps into 1D "Latent Horizontal Features" that capture the global layout context along the equator while bypassing vertical distortion entirely.
Similarly, Giovanni Pintore et al. introduced SliceNet: Deep Dense Depth Estimation From a Single Indoor Panorama Using a Slice-Based Representation, which slices the panorama vertically and feeds the sequence into a Long Short-Term Memory (LSTM) network. Because each slice represents a narrow vertical strip where distortion is uniform, the recurrent network can model horizontal relationships across the room without being disrupted by polar stretching. Zhijie Shen et al. expanded on this concept in Disentangling Orthogonal Planes for Indoor Panoramic Room Layout Estimation with Cross-Scale Distortion Awareness by explicitly separating vertical and horizontal structural features before sequence compression, ensuring the network remains aware of distortion across scales.
Strategy 4: Geometry-Aware Transformers
As the computer vision field transitions from CNNs to Vision Transformers (ViTs), the challenge of spherical distortion has shifted from local pixel windows to global attention mechanisms. Transformers do not have an inherent sense of spatial order; they rely entirely on position embeddings to understand where patches belong. In an ERP image, a patch at the equator covers a tiny physical area, while a patch at the pole covers a massive, sweeping arc.
To fix this, modern architectures bake 3D spherical coordinates directly into the transformer's attention mechanism. Xuewei Li et al. implemented this in SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation by introducing constrained patch embeddings and a latitude-weighted loss function, forcing the attention layers to discount the redundant, over-sampled information at the poles.
For large-scale foundation models, Haodong Li et al. developed DA$^{2}$: Depth Anything in Any Direction, deploying a specialized "SphereViT" backbone that computes attention based on the true angular distance between points on a sphere rather than their pixel distance on a flat grid. Ultimately, by decoupling the network from the 2D grid entirely, models like Pathformer3D: A 3D Scanpath Transformer for $360^{\circ }$ Images by Rong Quan et al. can operate directly within a native 3D spherical coordinate system, rendering the artifacts of flat projections completely obsolete.
Key Foundational Work
SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images (2018) — contributed 1 concept elements to this direction.
Kernel Transformer Networks for Compact Spherical Convolution (2019) — contributed 1 concept elements to this direction.
OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas (2018) — contributed 1 concept elements to this direction.