TL;DR
This paper introduces a deep learning method to estimate camera calibration parameters (pitch, roll, field of view, lens distortion) from a single image, outperforming previous single-image methods. Crucially, they also develop a novel "perceptual measure" based on human studies, showing their method's outputs are more visually pleasing and realistic for applications like virtual object insertion than those optimized solely for traditional error metrics.
What Problem Does This Paper Address?
Accurate camera calibration is essential for many computer vision and graphics tasks, including 3D scene reconstruction, augmented reality, and image editing. Traditionally, this requires tedious multi-image calibration procedures using specific targets. While single-image methods exist, they often rely on structured scenes or specific features. Furthermore, even when a camera is "accurately" calibrated according to standard metrics (like reprojection error), these calibrations might not be perceptually optimal for human observers, especially in applications like virtual object insertion where subtle geometric inaccuracies can break realism. This paper addresses the need for a fast, single-image calibration method that also considers human perception of geometric accuracy.
Methods
The core of the proposed method is a deep convolutional neural network (CNN) that takes a single image as input and outputs camera calibration parameters. Specifically, it estimates:
- Horizon line: Parameterized by its midpoint and angle (related to camera roll).
- Field of View (FoV): The effective horizontal field of view.
- Lens Distortion: Modeled using a unified spherical model, which is efficient and invertible.
The network is trained on a large, automatically generated dataset of images with corresponding ground truth camera parameters. A key innovation is the development of a perceptual measure. This was achieved by conducting large-scale human perception studies where participants judged the realism of virtual 3D objects inserted into images using either ground-truth or perturbed camera parameters. The results of these studies were used to build a quantitative measure of human sensitivity to calibration errors. The authors then evaluate their CNN against traditional metrics and this new perceptual measure.
Datasets
- Training/Validation/Test: A dataset of 205,865 image-parameter pairs was generated by extracting crops from 30,000 360° panoramas. Camera parameters (focal length, horizon, roll, aspect ratio, distortion) were sampled according to specific distributions to mimic real-world variations.
- Human Perception Studies: Two large-scale studies were conducted. Experiment 1 involved 530 panoramas and 10,368 images to evaluate sensitivity to pitch, roll, and FoV. Experiment 2 used 200 images to study sensitivity to lens distortion.
Key Insights and Ideas
- Perception vs. Accuracy: Minimizing traditional error metrics (like $\ell^2$ error) doesn't necessarily lead to the most perceptually plausible results for human observers. This is a critical insight for applications involving human interaction with computer-generated imagery.
- Human Sensitivity Varies: Humans are not equally sensitive to all types of calibration errors. For instance, they are more sensitive to roll errors when the image horizon is level and more sensitive to positive lens distortion errors.
- Unified Spherical Model: This model is effective for representing a wide range of camera distortions, including those from fisheye lenses, and is computationally efficient for both forward and inverse projection, making it suitable for deep learning.
- Emergent Feature Learning: The CNN, when analyzing images without clear vanishing lines, tends to focus on boundaries between sky and land for horizon estimation, demonstrating an emergent understanding of scene semantics.
Key Results
- The proposed deep learning method achieves competitive accuracy on standard metrics, outperforming previous single-image methods on their test set.
- Crucially, the method also significantly outperforms other approaches when evaluated using the newly developed perceptual measure. For example, their method's errors are "least noticeable" according to this measure.
- Human studies revealed specific sensitivities:
- High sensitivity to roll errors (errors of 12° or more are almost always detected).
- Tolerance for pitch errors up to 0.2 image units.
- Greater sensitivity to positive lens distortion errors.
- Sensitivity to distortion is higher in artificial/urban scenes than in nature scenes.
- The method was demonstrated to be effective for applications like 3D reconstruction, image undistortion, and image retrieval.
Limitations and Future Work
- Focal Length Ambiguity: The unified spherical model has an inherent ambiguity between focal length and distortion parameters, which could be addressed with a more compact representation.
- Extreme Roll/Pitch: The current data sampling and network might struggle with images exhibiting extreme roll (>$60^\circ$) or pitch (looking straight up/down).
- Principal Point Shift: Cropped images can break the radial symmetry assumption, leading to distortion estimation errors.
- Textureless Scenes: Images devoid of textures or recognizable scene elements pose challenges for accurate calibration.
- Future Work: The authors suggest exploring more compact parameter representations, incorporating domain knowledge into the CNN architecture, and investigating the use of their perceptual measure for training the network directly.
Summary updated