Tame a Wild Camera: In-the-Wild Monocular Camera Calibration

Shengjie Zhu, Abhinav Kumar, Masa Hu, Xiaoming Liu

Abstract

Camera calibration is a fundamental prerequisite for 3D computer vision, yet traditional methods rely on specific calibration patterns or rigid geometric assumptions that fail in "wild" scenarios. This paper introduces a novel approach to monocular camera calibration by learning a new 3D prior called the incidence field. By predicting the rays connecting 3D points to 2D pixels, the proposed method recovers full 4-Degree-of-Freedom (DoF) camera intrinsics from a single image. This approach is uniquely invariant to common image manipulations like cropping and resizing, enabling robust calibration of unconstrained internet photos and robotic captures.

Introduction

Determining internal camera parameters—specifically focal length and the principal point—is the "Day 0" requirement for mapping 2D pixels into 3D space. Traditionally, this is solved using checkerboard targets or by assuming a "Manhattan World" geometry where surfaces are perfectly orthogonal. However, the vast majority of images found online or captured by mobile robots do not satisfy these conditions. Furthermore, modern image processing often involves cropping and resizing, which shifts the principal point and alters the effective focal length, rendering standard calibration priors obsolete.

To address these challenges, this research proposes a shift from regressing numerical parameters to learning a geometric representation: the Incidence Field.

Methodology: The Incidence Field

Instead of attempting to directly predict the intrinsic matrix components ($f_x, f_y, c_x, c_y$), the authors learn an intermediate pixel-wise map.

1. Representation and Invariance

The incidence field is a map where each pixel stores a 3D unit vector representing the ray passing from the camera origin through that pixel. The core insight is that this field is invariant to cropping and resizing. While a crop changes the 2D coordinates of a pixel, the physical ray associated with that specific part of the scene remains constant. This local property makes the incidence field a robust bridge for calibrating edited images.

2. Learning the Prior

Mathematically, the incidence field is determined by the relationship between monocular depth maps and surface normals. However, deriving intrinsics directly from depth and normals is numerically unstable. The authors instead train a deep neural network (based on the NewCRFs architecture) to predict the incidence field directly from an undistorted RGB image. This creates a stable, learnable buffer that can handle noise better than raw geometric calculations.

3. The Geometric Solver

Once the network predicts the incidence field, the system employs a robust GPU-based RANSAC algorithm. This solver fits the best 4-DoF intrinsic matrix to the predicted rays, ensuring that the final camera model is geometrically consistent even if the network's initial pixel-wise predictions contain noise.

Experimental Setup and Datasets

To ensure the model generalizes across diverse environments, the researchers utilized a massive training suite comprising nine datasets, including:

  • Driving Scenes: NuScenes, KITTI, and Cityscapes.
  • Indoor Environments: NYUv2, ARKitScenes, and SUN3D.
  • Object-Centric Data: MVImgNet and Objectron.

During training, the authors synthesized "wild" intrinsics by applying random crops and resizes to the input images. Evaluation was conducted on "unseen" datasets to test zero-shot generalization, including MegaDepth (diverse internet photos), Waymo, and ScanNet.

Key Results and Impact

The proposed method demonstrates significant improvements over existing state-of-the-art (SOTA) calibration techniques:

  • Accuracy: On the GSV dataset, the method achieved a Mean Field of View (FoV) error of 2.47°, a notable improvement over the previous SOTA of 3.07°.
  • Zero-Shot Robustness: The model generalized effectively to MegaDepth, proving that it learns fundamental geometric priors rather than memorizing specific camera hardware.
  • Downstream Utility:
    • 3D Object Detection: Accurate intrinsics led to better 2D-to-3D projection of bounding boxes.
  • Image Restoration: The method can "undo" arbitrary crops, restoring images to their original centered focal point.
  • Pose Estimation: The system enabled superior two-view relative pose estimation for uncalibrated cameras, occasionally matching the performance of methods using ground-truth intrinsics.

Limitations and Future Work

While the incidence field approach is a significant step forward, it currently assumes "undistorted" images. It does not yet account for radial or fisheye distortion common in wide-angle lenses. Additionally, while the network inference is rapid, the RANSAC solver introduces computational overhead. Future research aims to integrate radial distortion estimation into the incidence field framework to provide a truly universal calibration tool for any camera lens.

Conclusion

By "taming" the wild camera through the incidence field, this work provides a robust solution for monocular calibration that thrives where traditional methods fail. Its invariance to image editing and its ability to generalize to unseen environments make it a powerful asset for 3D reconstruction, robotics, and large-scale vision tasks in the wild.

Summary updated

BibTeX
@inproceedings{zhu2023tame,
  author = {Shengjie Zhu and Abhinav Kumar and Masa Hu and Xiaoming Liu},
  title = {{Tame a Wild Camera: In-the-Wild Monocular Camera Calibration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=898RcRYWCg},
}
Figure