Most computer vision algorithms are built on a lie: the assumption that cameras behave like ideal pinhole models. In a pinhole world, straight lines remain straight, and an object’s appearance is "translation invariant"—meaning a car looks the same to a neural network whether it is in the center of the frame or near the edge.
However, wide-angle and fisheye lenses, essential for autonomous driving and 360° surveillance, violate these rules. They introduce extreme radial distortion, a non-linear warping where the magnification decreases with distance from the optical axis. As Tardif et al. note in Self-calibration of a General Radially Symmetric Distortion Model, standard models fail because these lenses capture fields of view exceeding 180°, turning straight world lines into deep curves.
The Failure of Translation Invariance
The core reason standard deep learning fails on fisheye imagery is the breakdown of the convolution operation's spatial assumptions. In a rectilinear image, a convolutional kernel learns a feature (like a vertical edge) that works everywhere. In a fisheye image, that same edge might appear vertical at the center but curved and tilted at the periphery.
Ravi Kumar et al. explain in SynDistNet: Self-Supervised Monocular Fisheye Camera Distance Estimation Synergized with Semantic Segmentation for Autonomous Driving that this violation of translation invariance causes monocular depth estimation to collapse. Because visual features change appearance based on their coordinates, a network trained on standard data cannot accurately "feel" distance in the warped periphery. Similarly, Duan et al. argue in RAPiD: Rotation-Aware People Detection in Overhead Fisheye Images that objects in overhead fisheye views change orientation based on their distance from the center, requiring models to be explicitly "rotation-aware" to maintain accuracy.
Rectification vs. Distortion Awareness
Researchers generally follow two paths to solve this: rectification (fixing the image) or distortion-aware learning (fixing the model).
1. Rectification and Flow
Traditional rectification uses mathematical models to "unwarp" pixels into a perspective view. However, this often introduces artifacts or requires precise calibration. To solve this, Lin and Yang proposed the Progressively Complementary Network (PCN), which uses "appearance flow" to learn the coordinate mapping required to fix shapes before the network processes them. Similarly, Shen et al. developed FishRecGAN, a Generative Adversarial Network that simultaneously rectifies distortion and calibrates camera parameters without needing manual checkerboard patterns.
2. Direct Learning in Distorted Space
The alternative is to teach the model to understand the distortion directly, avoiding the computational overhead of undistortion. Dias Pais et al. demonstrate this in OmniDRL by using Deep Reinforcement Learning to manipulate 3D bounding boxes in world coordinates. By bypassing the 2D image plane, the system detects pedestrians in 360° views without ever needing to "flatten" the image. Fanghui Zhang further refines this in Distorted Vehicle Detection and Distance Estimation by Metric Learning-Based SSD by using multi-scale partitioning to handle the varying resolution across the lens.
Mapping to Bird's Eye View
For autonomous tasks like valet parking, the most intuitive way to handle radial distortion is to project the warped pixels onto a ground plane. Shen and Shou describe this in Construction of Fisheye Lens Inverse Perspective Mapping Model and Its Applications of Obstacle Detection as a way to turn radial lines (which represent vertical obstacles) into a top-down map.
Modern approaches like F2BEV by Samani et al. take this further by using distortion-aware spatial cross-attention. Instead of a simple geometric projection, the model uses a transformer to "query" the distorted fisheye images, effectively learning which warped pixels correspond to specific locations in a geometrically consistent Bird's Eye View (BEV) map.
Universal Calibration
Underpinning all these methods is the need for a robust mathematical description of the lens itself. While the Rational Function model offers high precision, Yaroslava Lochman argues in BabelCalib: A Universal Approach to Calibrating Central Cameras that a "universal" framework is needed. By using intermediate back-projection models, it is possible to calibrate everything from standard pinholes to extreme catadioptric mirrors within a single linear solver, providing the geometric foundation necessary for the next generation of wide-angle vision.