A Deep Perceptual Measure for Lens and Camera Calibration

Yannick Hold-Geoffroy, Dominique Piché-Meunier, Kalyan Sunkavalli, Jean-Charles Bazin, François Rameau, Jean-François Lalonde

TL;DR

This paper introduces a novel deep learning approach for estimating camera and lens calibration parameters from a single image, moving beyond traditional, tedious multi-image methods. Crucially, it also develops a new "perceptual measure" of calibration accuracy based on human studies, demonstrating that minimizing standard error metrics isn't always optimal for realistic visual applications.

What Problem Does This Paper Address?

Accurately calibrating cameras (determining parameters like focal length, lens distortion, and orientation) is essential for many computer vision tasks, from 3D reconstruction to augmented reality and image editing. Traditional methods often require specialized calibration targets and multiple images, making them impractical for "in-the-wild" scenarios where we only have a single image and no prior knowledge of the camera. While deep learning has enabled single-image calibration, the paper questions whether minimizing standard error metrics (like reprojection error) truly leads to perceptually pleasing results for applications like virtual object insertion. They hypothesize that human perception of calibration errors might differ significantly from these objective metrics.

Methods

The core of the proposed method is a deep convolutional neural network (CNN) trained to estimate four camera parameters from a single image: horizon line (which implicitly defines pitch and roll), field of view (FoV), and lens distortion. The network is built upon a DenseNet architecture and uses a unified spherical model for camera projection, which is advantageous for its invertibility and ability to handle a wide range of distortions, including fisheye lenses.

To address the perceptual aspect, the authors conducted large-scale human perception studies. Participants were shown pairs of images with a virtual object inserted, one using ground-truth camera parameters and the other using perturbed parameters. They were asked to choose the more natural-looking insertion. This data was used to derive a novel "perceptual measure" that quantifies human sensitivity to different types and magnitudes of calibration errors. The CNN was then trained and evaluated using both standard error metrics and this new perceptual measure.

Datasets

The authors generated a large synthetic dataset for training their CNN. They leveraged 30,000 360° panoramas from the 360Cities dataset, extracting 7 rectified images from each. Camera parameters (focal length, horizon, roll, aspect ratio, and distortion) were sampled according to specific distributions designed to mimic real-world variations. This resulted in a dataset of over 200,000 image-parameter pairs. For evaluation, they used this synthetic dataset and also compared their method against existing single-image calibration methods on various real-world scenes.

Key Insights and Ideas

  • Beyond Pixel Error: The most significant insight is that minimizing pixel-level error doesn't always translate to human-perceived realism. This highlights the importance of perceptual studies in computer vision, especially for graphics and editing applications.
  • Human Sensitivity Varies: Humans are not equally sensitive to all types of calibration errors. For instance, roll errors are more easily detected than pitch errors, and large positive errors in FoV or roll are more noticeable than negative ones.
  • Context Matters for Distortion: Human sensitivity to lens distortion is influenced by the image content (more sensitive in urban scenes than nature) and the object's position.
  • Unified Spherical Model is Powerful: The choice of a unified spherical model for camera projection proved effective for handling a wide range of distortions and enabling efficient computation.
  • Single Image is Sufficient: The work demonstrates that a single image, combined with a well-trained deep network and a perceptual understanding, can yield robust camera calibration for various applications.

Key Results

  • The proposed deep learning method achieves competitive accuracy on standard error metrics compared to existing single-image calibration techniques.
  • Crucially, the method significantly outperforms previous approaches when evaluated using the newly developed perceptual measure, indicating that its estimated parameters lead to more visually plausible results for virtual object insertion.
  • The paper demonstrates the utility of their approach in several applications, including 3D reconstruction, image undistortion for "in-the-wild" images, and image retrieval based on camera parameters.
  • Human perception studies revealed specific sensitivities: roll errors are highly noticeable (±2.5° can go unnoticed), while pitch errors are more tolerable (up to 0.2 in image units).

Limitations and Future Work

  • Focal Length Ambiguity: The unified spherical model has an inherent ambiguity between focal length and distortion, which the authors acknowledge as a limitation. Future work could explore more compact representations to resolve this.
  • Extreme Camera Poses: The method struggles with images exhibiting extreme roll (over 60°) or pitch (looking straight up/down) due to limitations in their data sampling strategy.
  • Principal Point Shift: When images are cropped, the principal point shifts, breaking the radial symmetry assumption and leading to errors in distortion estimation.
  • Textureless Scenes: The method, like many geometry-based approaches, performs poorly on scenes lacking sufficient texture or recognizable elements.
  • Network Architecture: The authors suggest that a custom neural network architecture, rather than a generic one like DenseNet, might further improve performance by incorporating domain knowledge.
  • Perceptual Loss Training: While they experimented with training using their perceptual measure, it did not yield improvements over standard training, possibly because both losses converge to similar minima (zero error). Future work could explore this further.

Summary updated

Abstract

Image editing and compositing have become ubiquitous in entertainment, from digital art to AR and VR experiences. To produce beautiful composites, the camera needs to be geometrically calibrated, which can be tedious and requires a physical calibration target. In place of the traditional multi-image calibration process, we propose to infer the camera calibration parameters such as pitch, roll, field of view, and lens distortion directly from a single image using a deep convolutional neural network. We train this network using automatically generated samples from a large-scale panorama dataset, yielding competitive accuracy in terms of standard `2 error. However, we argue that minimizing such standard error metrics might not be optimal for many applications. In this work, we investigate human sensitivity to inaccuracies in geometric camera calibration. To this end, we conduct a large-scale human perception study where we ask participants to judge the realism of 3D objects composited with correct and biased camera calibration parameters. Based on this study, we develop a new perceptual measure for camera calibration and demonstrate that our deep calibration network outperforms previous single-image based calibration methods both on standard metrics as well as on this novel perceptual measure. Finally, we demonstrate the use of our calibration network for several applications, including virtual object insertion, image retrieval, and compositing. A demonstration of our approach is available at https://lvsn.github.io/deepcalib .
BibTeX
@article{holdgeoffroy2023deep,
  author = {Yannick Hold-Geoffroy and Dominique Piché-Meunier and Kalyan Sunkavalli and Jean-Charles Bazin and François Rameau and Jean-François Lalonde},
  title = {{A Deep Perceptual Measure for Lens and Camera Calibration}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=53Gs82o7Mr},
}
Figure