Deep geometry-aware camera self-calibration from video

Annika Hagemann, Moritz Knorr, Christoph Stiller

Deep geometry-aware camera self-calibration from video

Introduction

To perceive the world in 3D, computer vision systems require knowledge of "intrinsics"—the mathematical mapping between 3D points in space and 2D pixels on a sensor. Traditionally, this necessitates a tedious target-based calibration process using checkerboards. While "self-calibration" (inferring these parameters from arbitrary scenes) exists, it faces a significant trade-off. Classical methods generalize well but struggle with complex feature matching in "in the wild" videos. Conversely, deep learning methods excel at feature matching but often treat calibration as a simple regression problem, failing to generalize to unseen cameras or environments.

DroidCalib bridges this gap by combining the robust feature-learning capabilities of neural networks with the mathematical rigor of classical geometry.

Methodology

DroidCalib is built upon the DROID-SLAM architecture, introducing a novel Self-Calibrating Bundle Adjustment (SC-BA) layer.

Overview of DroidCalib. Top: Geometric intuition showing how different intrinsics affect point projection. Bottom: The end-to-end pipeline with the differentiable SC-BA layer.

The core innovations include:

  • Explicit Geometry: Instead of regressing focal lengths directly, the model explicitly utilizes projection equations (e.g., Pinhole or Unified Camera Models). This ensures the physics of light projection is respected.
  • The SC-BA Layer: This differentiable optimization layer takes weighted feature correspondences predicted by a CNN and performs several Gauss-Newton steps to jointly optimize camera poses, pixel depths, and camera intrinsics ($\theta$).
  • Differentiable Optimization: The entire process is formulated to be differentiable, allowing for end-to-end training. The network learns to predict correspondences and confidence weights that specifically optimize the calibration process.
  • Model Agnosticism: By explicitly modeling the projection function, the system can adapt to different camera types, such as wide-angle fisheye lenses, by simply updating the mathematical model in the SC-BA layer without retraining the neural network.

Experimental Evaluation

The model was evaluated on several benchmarks, including TartanAir (synthetic), EuRoC MAV (real-world drone), and TUM RGB-D (handheld).

  • Superior Accuracy: DroidCalib outperformed existing baselines like COLMAP combined with SuperPoint/SuperGlue. On TartanAir, it reduced median mapping error by 49% compared to the next best method.
  • Distortion Handling: On raw EuRoC data with heavy radial distortion, DroidCalib achieved a median mapping error of just 0.40 pixels, significantly outperforming other self-calibration methods.
  • Robustness: The system can recover correct parameters even from a 25% initial error in focal length, maintaining accurate trajectories where standard deep SLAM systems fail.
  • Efficiency: Running at approximately 4 fps on a single GPU, it is faster than many classical SfM pipelines, though slightly slower than the original DROID-SLAM (10 fps).

Discussion and Limitations

While DroidCalib demonstrates significant progress, it faces challenges with "degenerate motion" (e.g., pure rotation), where geometry becomes mathematically ambiguous. Additionally, hardware artifacts like rolling shutter effects and motion blur in the TUM dataset impacted accuracy, as these are not yet accounted for in the geometric model. Future work will focus on addressing these artifacts and optimizing the memory-intensive SC-BA layer, which currently creates dense blocks in the Hessian matrix due to the global nature of intrinsics.

Summary updated

Abstract

Accurate intrinsic calibration is essential for camera-based 3D perception, yet, it typically requires targets of well-known geometry. Here, we propose a camera self-calibration approach that infers camera intrinsics during application, from monocular videos in the wild. We propose to explicitly model projection functions and multi-view geometry, while leveraging the capabilities of deep neural networks for feature extraction and matching. To achieve this, we build upon recent research on integrating bundle adjustment into deep learning models, and introduce a self-calibrating bundle adjustment layer. The self-calibrating bundle adjustment layer optimizes camera intrinsics through classical Gauß-Newton steps and can be adapted to different camera models without re-training. As a specific realization, we implemented this layer within the deep visual SLAM system DROID-SLAM, and show that the resulting model, DroidCalib, yields state-of-the-art calibration accuracy across multiple public datasets. Our results suggest that the model generalizes to unseen environments and different camera models, including significant lens distortion. Thereby, the approach enables performing 3D perception tasks without prior knowledge about the camera. Code is available at https://github.com/boschresearch/droidcalib.
BibTeX
@inproceedings{hagemann2023deep,
  author = {Annika Hagemann and Moritz Knorr and Christoph Stiller},
  title = {{Deep geometry-aware camera self-calibration from video}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00318},
}
Figure