A Perceptual Measure for Deep Single Image Camera and Lens Calibration

Yannick Hold-Geoffroy, Dominique Piché-Meunier, Kalyan Sunkavalli, Jean-Charles Bazin, François Rameau, Jean-François Lalonde

TL;DR

This paper introduces a deep learning method to estimate camera calibration parameters (pitch, roll, field of view, lens distortion) from a single image, outperforming previous single-image methods. Crucially, they also develop a novel "perceptual measure" based on human studies, showing their method's outputs are more visually pleasing and realistic for applications like virtual object insertion than those optimized solely for traditional error metrics.

What Problem Does This Paper Address?

Accurate camera calibration is essential for many computer vision and graphics tasks, including 3D scene reconstruction, augmented reality, and image editing. Traditionally, this requires tedious multi-image calibration procedures using specific targets. While single-image methods exist, they often rely on structured scenes or specific features. Furthermore, even when a camera is "accurately" calibrated according to standard metrics (like reprojection error), these calibrations might not be perceptually optimal for human observers, especially in applications like virtual object insertion where subtle geometric inaccuracies can break realism. This paper addresses the need for a fast, single-image calibration method that also considers human perception of geometric accuracy.

Methods

The core of the proposed method is a deep convolutional neural network (CNN) that takes a single image as input and outputs camera calibration parameters. Specifically, it estimates:

  • Horizon line: Parameterized by its midpoint and angle (related to camera roll).
  • Field of View (FoV): The effective horizontal field of view.
  • Lens Distortion: Modeled using a unified spherical model, which is efficient and invertible.

The network is trained on a large, automatically generated dataset of images with corresponding ground truth camera parameters. A key innovation is the development of a perceptual measure. This was achieved by conducting large-scale human perception studies where participants judged the realism of virtual 3D objects inserted into images using either ground-truth or perturbed camera parameters. The results of these studies were used to build a quantitative measure of human sensitivity to calibration errors. The authors then evaluate their CNN against traditional metrics and this new perceptual measure.

Datasets

  • Training/Validation/Test: A dataset of 205,865 image-parameter pairs was generated by extracting crops from 30,000 360° panoramas. Camera parameters (focal length, horizon, roll, aspect ratio, distortion) were sampled according to specific distributions to mimic real-world variations.
  • Human Perception Studies: Two large-scale studies were conducted. Experiment 1 involved 530 panoramas and 10,368 images to evaluate sensitivity to pitch, roll, and FoV. Experiment 2 used 200 images to study sensitivity to lens distortion.

Key Insights and Ideas

  • Perception vs. Accuracy: Minimizing traditional error metrics (like $\ell^2$ error) doesn't necessarily lead to the most perceptually plausible results for human observers. This is a critical insight for applications involving human interaction with computer-generated imagery.
  • Human Sensitivity Varies: Humans are not equally sensitive to all types of calibration errors. For instance, they are more sensitive to roll errors when the image horizon is level and more sensitive to positive lens distortion errors.
  • Unified Spherical Model: This model is effective for representing a wide range of camera distortions, including those from fisheye lenses, and is computationally efficient for both forward and inverse projection, making it suitable for deep learning.
  • Emergent Feature Learning: The CNN, when analyzing images without clear vanishing lines, tends to focus on boundaries between sky and land for horizon estimation, demonstrating an emergent understanding of scene semantics.

Key Results

  • The proposed deep learning method achieves competitive accuracy on standard metrics, outperforming previous single-image methods on their test set.
  • Crucially, the method also significantly outperforms other approaches when evaluated using the newly developed perceptual measure. For example, their method's errors are "least noticeable" according to this measure.
  • Human studies revealed specific sensitivities:
  • High sensitivity to roll errors (errors of 12° or more are almost always detected).
  • Tolerance for pitch errors up to 0.2 image units.
  • Greater sensitivity to positive lens distortion errors.
  • Sensitivity to distortion is higher in artificial/urban scenes than in nature scenes.
  • The method was demonstrated to be effective for applications like 3D reconstruction, image undistortion, and image retrieval.

Limitations and Future Work

  • Focal Length Ambiguity: The unified spherical model has an inherent ambiguity between focal length and distortion parameters, which could be addressed with a more compact representation.
  • Extreme Roll/Pitch: The current data sampling and network might struggle with images exhibiting extreme roll (>$60^\circ$) or pitch (looking straight up/down).
  • Principal Point Shift: Cropped images can break the radial symmetry assumption, leading to distortion estimation errors.
  • Textureless Scenes: Images devoid of textures or recognizable scene elements pose challenges for accurate calibration.
  • Future Work: The authors suggest exploring more compact parameter representations, incorporating domain knowledge into the CNN architecture, and investigating the use of their perceptual measure for training the network directly.

Summary updated

Abstract

Image editing and compositing have become ubiquitous in entertainment, from digital art to AR and VR experiences. To produce beautiful composites, the camera needs to be geometrically calibrated, which can be tedious and requires a physical calibration target. In place of the traditional multi-image calibration process, we propose to infer the camera calibration parameters such as pitch, roll, field of view, and lens distortion directly from a single image using a deep convolutional neural network. We train this network using automatically generated samples from a large-scale panorama dataset, yielding competitive accuracy in terms of standard <inline-formula><tex-math notation="LaTeX">$\ell ^{2}$</tex-math></inline-formula> error. However, we argue that minimizing such standard error metrics might not be optimal for many applications. In this work, we investigate human sensitivity to inaccuracies in geometric camera calibration. To this end, we conduct a large-scale human perception study where we ask participants to judge the realism of 3D objects composited with correct and biased camera calibration parameters. Based on this study, we develop a new perceptual measure for camera calibration and demonstrate that our deep calibration network outperforms previous single-image based calibration methods both on standard metrics as well as on this novel perceptual measure. Finally, we demonstrate the use of our calibration network for several applications, including virtual object insertion, image retrieval, and compositing.
BibTeX
@article{holdgeoffroy2023perceptual,
  author = {Yannick Hold-Geoffroy and Dominique Piché-Meunier and Kalyan Sunkavalli and Jean-Charles Bazin and François Rameau and Jean-François Lalonde},
  title = {{A Perceptual Measure for Deep Single Image Camera and Lens Calibration}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=53Gs82o7Mr},
}

Loading related content...

Figure