Deep geometry-aware camera self-calibration from video
Introduction
To perceive the world in 3D, computer vision systems require knowledge of "intrinsics"—the mathematical mapping between 3D points in space and 2D pixels on a sensor. Traditionally, this necessitates a tedious target-based calibration process using checkerboards. While "self-calibration" (inferring these parameters from arbitrary scenes) exists, it faces a significant trade-off. Classical methods generalize well but struggle with complex feature matching in "in the wild" videos. Conversely, deep learning methods excel at feature matching but often treat calibration as a simple regression problem, failing to generalize to unseen cameras or environments.
DroidCalib bridges this gap by combining the robust feature-learning capabilities of neural networks with the mathematical rigor of classical geometry.
Methodology
DroidCalib is built upon the DROID-SLAM architecture, introducing a novel Self-Calibrating Bundle Adjustment (SC-BA) layer.
The core innovations include:
- Explicit Geometry: Instead of regressing focal lengths directly, the model explicitly utilizes projection equations (e.g., Pinhole or Unified Camera Models). This ensures the physics of light projection is respected.
- The SC-BA Layer: This differentiable optimization layer takes weighted feature correspondences predicted by a CNN and performs several Gauss-Newton steps to jointly optimize camera poses, pixel depths, and camera intrinsics ($\theta$).
- Differentiable Optimization: The entire process is formulated to be differentiable, allowing for end-to-end training. The network learns to predict correspondences and confidence weights that specifically optimize the calibration process.
- Model Agnosticism: By explicitly modeling the projection function, the system can adapt to different camera types, such as wide-angle fisheye lenses, by simply updating the mathematical model in the SC-BA layer without retraining the neural network.
Experimental Evaluation
The model was evaluated on several benchmarks, including TartanAir (synthetic), EuRoC MAV (real-world drone), and TUM RGB-D (handheld).
- Superior Accuracy: DroidCalib outperformed existing baselines like COLMAP combined with SuperPoint/SuperGlue. On TartanAir, it reduced median mapping error by 49% compared to the next best method.
- Distortion Handling: On raw EuRoC data with heavy radial distortion, DroidCalib achieved a median mapping error of just 0.40 pixels, significantly outperforming other self-calibration methods.
- Robustness: The system can recover correct parameters even from a 25% initial error in focal length, maintaining accurate trajectories where standard deep SLAM systems fail.
- Efficiency: Running at approximately 4 fps on a single GPU, it is faster than many classical SfM pipelines, though slightly slower than the original DROID-SLAM (10 fps).
Discussion and Limitations
While DroidCalib demonstrates significant progress, it faces challenges with "degenerate motion" (e.g., pure rotation), where geometry becomes mathematically ambiguous. Additionally, hardware artifacts like rolling shutter effects and motion blur in the TUM dataset impacted accuracy, as these are not yet accounted for in the geometric model. Future work will focus on addressing these artifacts and optimizing the memory-intensive SC-BA layer, which currently creates dense blocks in the Hessian matrix due to the global nature of intrinsics.
Summary updated