Abstract
Light field essentially represents the collection of rays in space. The rays captured by multiple light field cameras form subsets of full rays in 3D space and can be transformed to each other. However, most previous approaches model the projection from an arbitrary point in 3D space to corresponding pixel on the sensor. There are few models on describing the ray sampling and transformation among multiple light field cameras. In the paper, we propose a novel ray-space projection model to transform sets of rays captured by multiple light field cameras in term of the Plucker coordinates. We first derive a 6x6 ray-space intrinsic matrix based on multi-projection-center (MPC) model. A homogeneous ray-space projection matrix and a fundamental matrix are then proposed to establish ray-ray correspondences among multiple light fields. Finally, based on the ray-space projection matrix, a novel camera calibration method is proposed to verify the proposed model. A linear constraint and a ray-ray cost function are established for linear initial solution and non-linear optimization respectively. Experimental results on both synthetic and real light field data have verified the effectiveness and robustness of the proposed model.
Introduction
Traditional light field camera models usually focus on the projection from a 3D point to a specific pixel on a sensor. While this works for single-camera setups, it becomes complex when multiple light field cameras are involved or when "stitching" different light fields together. The core challenge is that light fields are fundamentally about rays, not just points. Existing models often have high redundancy or struggle to define how a ray captured by one camera relates to a ray captured by another. Without a generalized geometric model for ray sampling and transformation, tasks like depth estimation across wide baselines or multi-camera registration remain difficult and prone to error.
Methodology
The proposed model shifts the perspective from "point-to-pixel" to "ray-to-ray" using several key innovations:
- Plücker Coordinates: Instead of standard Euclidean coordinates, rays are represented using Plücker line coordinates (moment and direction). This provides a homogeneous parameterization that makes rotation and translation of rays mathematically straightforward.
- Ray-Space Intrinsic Matrix (RSIM): A $6 \times 6$ matrix ($K$) relates recorded pixel indices $(i, j, u, v)$ directly to physical rays in 3D space. Unlike previous 12-parameter models, this RSIM uses only 6 independent parameters, making it more stable and analogous to the intrinsic matrix used in traditional computer vision.
- Ray-Space Projection & Fundamental Matrices: A homogeneous projection matrix ($P$) maps rays between world and camera coordinates, and a Fundamental Matrix ($F$) establishes correspondences between two different light field cameras.
To verify the model, a calibration algorithm is proposed. It starts with a linear initial solution using a checkerboard and follows up with non-linear optimization using a custom "ray-ray cost function" that minimizes the geometric distance between rays and the lines on the calibration board.
Experimental Results
The researchers evaluated their model using both simulated and real-world data. Synthetic data mimicking a Lytro camera was used to test robustness against varying noise levels (0.1 to 1.5 pixels). Real-world data included standard datasets (DPW, MPC) and raw data from Lytro and Lytro Illum cameras.
Robustness and Accuracy
The model maintained high accuracy even with high image noise. Relative errors for intrinsic parameters stayed below 0.5% at a noise level of 0.5 pixels.
In comparative tests against state-of-the-art methods (DPW and BJW), the proposed model consistently achieved lower Root Mean Square (RMS) ray re-projection errors. On several real datasets, the method reduced mean re-projection error to roughly 0.10–0.18 pixels.
Practical Applications
The authors demonstrated successful light field stitching and highly accurate 3D reconstruction. Distance measurements between points in a reconstructed scene had relative errors as low as 0.10% to 1.57% compared to physical measurements.
Conclusion
By using the Multi-Projection-Center (MPC) model as a base and adopting a "ray-ray" philosophy, the authors eliminated parameter redundancy and improved optimization convergence. The RSIM allows researchers to use familiar concepts like the Fundamental Matrix in light field geometry. Future work will address complex lens distortions and scaling the model to large-scale camera arrays.
Summary updated