Standard computer vision algorithms are built on a lie: the assumption that the world is projected onto a flat, infinite plane. This "pinhole" model works well for narrow-view cameras, but it fails catastrophically for the wide-angle, fisheye, and 360-degree sensors used in robotics and autonomous driving. When a field of view approaches or exceeds 180 degrees, the math of flat planes breaks—straight lines become extreme curves, and objects stretch or compress depending on their distance from the lens center.
To solve this, researchers have converged on a more robust geometric foundation: mapping visual data onto a unit sphere. By treating every pixel as a 3D direction vector rather than a 2D coordinate, we can create a "lens-agnostic" framework where the same math applies to a standard GoPro, a fisheye security camera, or a complex mirror-based catadioptric system.
The Problem with Rectification
The traditional way to handle lens distortion is "undistortion" or rectification—warping the image until it looks like a pinhole projection. However, as Jaouad Hajjami and colleagues point out in ArUcOmni, this process is destructive. Rectification often crops out valuable data at the edges or introduces interpolation artifacts that destroy fine details.
More importantly, a pinhole projection cannot mathematically represent anything "behind" the camera. For ultra-wide sensors, this means throwing away half the available information. By moving to a spherical model, systems like FisheyeBEVSeg can map pixels directly to 3D direction vectors, preserving the full field of view without the resolution loss inherent in flattening the image.
Space-Invariance on the Sphere
In a standard image, an object’s appearance changes as it moves from the center to the edge due to radial distortion. This makes feature matching difficult. Peter Hansen and Peter Corke addressed this in Scale-Invariant Features on the Sphere by redefining the SIFT algorithm for spherical geometry.
The intuition is elegant: on a sphere, a camera rotation is just a rotation of the signal. By performing scale-space analysis using the spherical diffusion equation (implemented via spherical harmonics) rather than standard Gaussian blur, feature detection becomes "space-invariant." A feature looks the same to the algorithm whether it is directly in front of the lens or at the extreme periphery. This allows for significantly more stable tracking in robotic navigation, as demonstrated in Scale invariant feature matching with wide angle images.
A Unified Geometric Frontend
If we can map any camera's pixels to a sphere, we can build a single vision pipeline that works for any hardware. Peter F. Sturm and João P. Barreto formalized this in General Imaging Geometry for Central Catadioptric Cameras. They showed that complex mirror-lens systems can be modeled as a two-stage process: project 3D points onto a unit sphere, then map that sphere to the image plane.
This abstraction enables "lens-agnostic" deep learning. The Unified Spherical Frontend (USF), developed by Mukai Yu and others, uses this principle to achieve rotation equivariance. Because the model operates in the spherical domain using distance-based kernels, it maintains high performance even if the input image is captured by an entirely different lens type than the one used during training. Similarly, the Position-Invariant Transform (PIT) by Qianyu Zhou uses spherical projection to normalize perspective deformations, allowing models to generalize across different Fields of View (FoV).
Handling the Horizon and Beyond
Spherical representations are particularly powerful for 3D reconstruction and odometry. In Omnidirectional DSO, Hidenobu Matsuki and Daniel Cremers show that using a unified omnidirectional model allows a system to track motion using the full 180°+ view.
Traditional bundle adjustment often struggles with points at the horizon because their depth is effectively infinite. Frank Schindler and Thomas Läbe solved this in their BACS framework by switching from Euclidean normalization to spherical normalization. On a sphere, a point at infinity is just a well-defined direction vector. This allows the system to use distant landmarks to stabilize rotation estimates, even when those landmarks are "behind" the camera's forward-facing center.
Conclusion
By moving from the flat image plane to the unit sphere, researchers have moved away from fixing distortions and toward embracing the true geometry of light. Whether it is LF-PGVIO tracking geodesic segments on a sphere or Fisheye-GS rendering 3D scenes directly from fisheye data, the shift toward spherical representations provides a mathematically consistent, information-preserving, and hardware-independent foundation for modern computer vision.