Modeling Imaging Systems as Bundles of Rays

113 concepts80 papers1 proposal

Traditional computer vision relies heavily on the pinhole camera model. This elegant abstraction assumes that all light rays passing through a scene intersect at a single point—the optical center—before projecting onto a flat image plane. To account for real-world lens imperfections, engineers typically append parametric distortion formulas, such as Taylor series polynomials, to warp the coordinates back into a perspective framework.

However, this paradigm breaks down when confronted with modern, complex, or highly distorted optical systems. Extreme fisheye lenses, catadioptric (mirror-based) systems, underwater housings causing irregular refraction, and multi-camera rigs do not possess a single, clean center of projection. For these architectures, fitting a standard polynomial model introduces severe mathematical friction. As noted by Fan et al. in SIR, relying on rigid analytical warping functions limits our ability to handle specialized optics that deviate from standard geometric formulas.

The solution is to discard the pinhole assumption entirely and adopt a generalized or ray-based camera model. Instead of forcing a global algebraic formula onto the camera hardware, this approach treats the imaging system as a "black box" that maps each individual pixel coordinate to a unique 3D light ray in space. As formalized by Sturm in Multi-View Geometry for General Camera Models, this abstraction unifies all imaging systems under a single geometric language, transforming computer vision from a study of lens-specific equations into a pure study of ray intersections.


The Geometry of Unconstrained Rays

Why does treating pixels as independent rays simplify 3D reconstruction and motion estimation? The core benefit is that it completely decouples the hardware-specific optics from the downstream geometric optimization. Once every pixel is translated into a 3D line (defined by an origin and a direction), fundamental tasks like triangulation or bundle adjustment become invariant to the underlying hardware.

Traditional Pinhole:      Pixel (x,y) ---> [ Polynomial Formula ] ---> Single Optical Center
Generalized Camera:       Pixel (x,y) ---> Independent 3D Ray (Origin + Direction Vector)

In a standard central camera, all ray origins happen to coincide at the same point. In non-central cameras, they do not. For instance, consider a self-driving car equipped with a ring of overlapping cameras. Instead of treating each sensor as an independent entity and constantly managing the coordinate transformations between them, Lee et al. demonstrate in Motion Estimation for Self-Driving Cars with a Generalized Camera that the entire multi-camera rig can be modeled as a single generalized camera. By representing the light rays across all sensors as Plücker lines, the system can perform unified motion estimation.

This abstraction yields a massive mathematical advantage regarding scale. In a single perspective camera, scene reconstruction is always bound by an ambiguous scale factor; you cannot distinguish a small object close up from a massive object far away. However, because a generalized camera model naturally incorporates the physical offsets (baselines) between the rays of different sensors, it inherently encodes metric scale. As shown by Lee et al. in Self-Calibration and Visual SLAM with a Multi-Camera System on a Micro Aerial Vehicle, this allows a multi-camera system to recover its exact trajectory and metric scale without relying on external sensors like LiDAR or GPS.

This concept scales up seamlessly. Entire collections of previously registered images or view-graphs can be treated as a single virtual generalized camera. As demonstrated by Zheng et al. in Structure from Motion Using Structure-Less Resection and Kneip et al. in The generalized relative pose and scale problem: View-graph fusion via 2D-2D registration, this abstraction allows new images or separate map segments to be registered directly to an existing model via ray intersections, bypassing the need to explicitly triangulate noisy 3D point clouds.


Non-Parametric and Learned Ray Mappings

If we abandon fixed algebraic formulas, how do we practically map an image's pixel grid to these 3D rays? The field has evolved from discrete lookup tables to continuous, learned representations.

Discrete and Spline-Based Representations

The simplest non-parametric approach is to treat the distortion as a data-driven curve or lookup table. Hartley et al. demonstrated in Parameter-Free Radial Distortion Correction with Centre of Distortion Estimation that treating a distortion curve as a discrete numerical mapping allows a model to adapt to extreme wide-angle and fisheye lenses without the risk of model mismatch or overfitting. To enforce continuity and invertibility across the sensor plane, Wang proposed in Structure-from-Motion with a Non-Parametric Camera Model to model the relationship between incident light ray angles and image sensor radii using natural cubic splines. This provides a smooth, flexible mapping that can accommodate highly irregular lens geometries during an incremental Structure-from-Motion (SfM) pipeline.

Neural Ray Fields

With the rise of differentiable rendering, researchers have begun training neural networks to act as the coordinate-to-ray mapping function. Vasiljevic et al. introduced Neural Ray Surfaces (NRS) in Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion, where a neural network maps each 2D pixel coordinate directly to a 3D unit vector representing the direction of light. Because the network is fully differentiable, it can be optimized using self-supervised photometric consistency losses.

Pixel Coordinate (u, v) ---> [ Neural Network / MLP ] ---> 3D Ray Direction (dx, dy, dz)

This allows a system to learn the camera's internal geometry and the scene's 3D structure simultaneously from video sequences, making it completely agnostic to whether the video was shot on a standard lens, a fisheye lens, or through a highly refractive underwater port. Similarly, Jeong et al. in Self-Calibrating Neural Radiance Fields demonstrated that incorporating per-pixel differentiable camera models directly into a radiance field optimization enables high-fidelity 3D reconstructions from uncalibrated, highly distorted imagery by treating lens distortion as a learnable component of the scene.


Calibrating the Black Box

Calibrating a camera that has no predefined mathematical model sounds paradoxically difficult. If every pixel can map to any arbitrary ray, an unconstrained optimization would immediately overfit or fail to converge without a perfect initialization.

To solve this, modern frameworks employ a two-stage or intermediate calibration strategy. Instead of mapping pixels directly to final parameters, they introduce a model-agnostic intermediate representation. For example, Porfiri Dal Cin et al. in Revisiting Calibration of Wide-Angle Radially Symmetric Cameras decoupled the estimation of physical light-ray directions from the fitting of specific camera model parameters. By training a neural network to first predict a model-agnostic viewing-angle representation, they can subsequently fit target parameters via robust optimization, avoiding the initialization failures that plague end-to-end networks.

A similar philosophy underpins BabelCalib: A Universal Approach to Calibrating Central Cameras. Developed by Lochman et al., this framework uses a flexible intermediate back-projection model as a mathematical bridge. The pipeline first solves for geometric poses and ray directions using linear solvers, and then uses a model-to-model regression strategy to map this generic intermediate representation into whatever target parametric model (such as OpenCV's fisheye or a catadioptric formula) the user desires.

Even when we want to enforce specific physical constraints—such as requiring all rays to intersect at a single optical center to maintain compatibility with standard epipolar geometry—we do not need to revert to rigid polynomials. Bergamasco et al. showed in Parameter-Free Lens Distortion Calibration of Central Cameras that this centrality constraint can be solved as a generalized least-squares problem over a non-parametric model. This allows the camera profile to remain completely flexible and data-driven while mathematically guaranteeing that all recovered rays intersect at a single common point.

By treating cameras as bundles of rays rather than collections of equations, the generalized camera model provides a mathematically elegant, hardware-agnostic foundation that allows computer vision algorithms to see across any optical medium.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 77 papers
  2. Version 1 · 2026-10-04 · 79 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_generalized_camera_model_73ddf817,
  title = {generalized camera model},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalized-camera-model/rdr_73ddf817cf264b51b6dc5a6db6eb616c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_generalized_camera_model_73ddf817_v2,
  title = {generalized camera model},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalized-camera-model/rdr_73ddf817cf264b51b6dc5a6db6eb616c/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake