DiffCalib: Reformulating Monocular Camera Calibration as Diffusion-Based Dense Incident Map Generation

Xiankang He, Guangkai Xu, Bo Zhang, Hao Chen, Ying Cui, Dongyan GuoAAAI 2025 · DOI

TL;DR

This paper introduces DiffCalib, a novel method for monocular camera calibration that reformulates the problem as generating a "dense incident map" using pre-trained diffusion models. By leveraging the rich visual priors of diffusion models and jointly estimating depth, DiffCalib achieves state-of-the-art accuracy and robustness in challenging "in-the-wild" scenarios, outperforming existing methods by up to 40%.

What Problem Does This Paper Address?

Accurately determining a camera's intrinsic parameters (like focal length and optical center) is crucial for many 3D computer vision tasks, such as reconstructing scenes from images. Traditional methods often rely on specific assumptions about the scene (e.g., Manhattan world) or require controlled environments and specific calibration objects (like checkerboards). These limitations make them struggle to generalize to real-world, unconstrained "in-the-wild" images. While recent approaches have tried to alleviate these issues by predicting an "incident field" (rays from 3D points to 2D pixels), they still face challenges due to insufficient training data and the inherent ill-posed nature of monocular depth estimation. This paper aims to bridge this gap by developing a more robust and generalizable monocular camera calibration method that works effectively even with single, uncalibrated images from diverse real-world scenes.

Methods

DiffCalib's core innovation lies in reformulating camera calibration as a dense incident map generation task. Instead of directly predicting the four intrinsic parameters, the model learns to generate an "incident map," which essentially details the angle of incidence for each pixel in the image. This map's format is well-suited for diffusion models.

The pipeline leverages a pre-trained Stable Diffusion v2.1 model. Specifically, the authors freeze the Variational Autoencoder (VAE) encoder and decoder and fine-tune the U-Net component. The U-Net is trained to predict the noise added to the latent representation of the incident map.

A key enhancement is the joint estimation of the incident map and a depth map. This is motivated by the observed correlation between camera intrinsics and depth information. By learning both simultaneously, the model benefits from the complementary geometric cues, leading to improved performance for both tasks.

During inference, the model generates multiple incident maps by sampling different random noises and then averages them to improve robustness. The camera intrinsic parameters are then derived from the generated incident map using a non-learning-based RANSAC algorithm, which is a standard technique for robustly fitting models to data.

Datasets

  • Training: Hypersim (synthetic indoor scenes with ground truth intrinsics and depth) was used as the primary training dataset. To increase variety, additional datasets like NuScenes, KITTI, CityScapes, NYUv2, SUN3D, and ARKitScenes were incorporated, using their camera intrinsic parameters but without depth information.
  • Testing: For monocular camera calibration, evaluation was performed on datasets such as Waymo, RGBD, ScanNet, MVS, and Scenes11. For 3D reconstruction, NYUv2 and ETH3D were used.

Key Insights and Ideas

  • Diffusion Models for Calibration: The paper demonstrates that the rich visual knowledge embedded in pre-trained diffusion models can be effectively transferred to perception tasks beyond image generation, specifically to camera intrinsic estimation.
  • Incident Map as a Diffusion-Friendly Representation: Reformulating calibration as incident map generation aligns the problem with the strengths of diffusion models, allowing them to leverage their learned priors for robust estimation.
  • Joint Depth and Incident Map Estimation: The inherent relationship between depth and camera intrinsics is exploited by jointly learning both. This synergistic approach significantly boosts the performance of both tasks.
  • Robustness through Ensemble: Generating multiple incident maps from different noise samples and averaging them leads to more confident and reliable estimates, especially in challenging real-world scenarios.
  • Generalization to "In-the-Wild" Scenes: By moving away from specific assumptions and objects, and by leveraging powerful pre-trained models, DiffCalib achieves strong generalization capabilities on diverse, uncalibrated datasets.

Key Results

  • State-of-the-Art Calibration: DiffCalib achieves state-of-the-art performance on monocular camera calibration across multiple datasets, outperforming existing methods by up to a 40% reduction in prediction errors.
  • Improved Generalization: The method shows superior performance on zero-shot "in-the-wild" datasets, demonstrating its robustness and ability to handle unconstrained scenarios.
  • Enhanced 3D Reconstruction: The accurate camera intrinsics and depth maps generated by DiffCalib significantly benefit downstream 3D reconstruction tasks, showing improved geometric accuracy and detail preservation compared to methods like LeReS.
  • Efficiency with Limited Data: Even when trained with less data (e.g., 48k samples instead of 80k), DiffCalib still outperforms previous methods, highlighting the effectiveness of fine-tuning large pre-trained models.

Limitations and Future Work

  • RGBD Discrepancy: The paper notes that DiffCalib's performance on RGBD and Scenes11 datasets is slightly lower, potentially due to uncalibrated default intrinsic parameters in RGBD and the complexity of Scenes11's objects. The authors suggest that in these specific cases, using only incident maps without depth estimation might be more beneficial.
  • Affine-Invariant Depth: The generated depth map is affine-invariant, requiring an additional step (using a frozen shift model) to recover the correct 3D point cloud for reconstruction.
  • Future Work: The authors suggest that the robust performance of DiffCalib can greatly benefit other downstream applications beyond 3D scene reconstruction. The paper also implicitly points towards further exploration of how diffusion models can be adapted for various geometric estimation tasks.

Summary updated

BibTeX
@inproceedings{he2025diffcalib,
  author = {Xiankang He and Guangkai Xu and Bo Zhang and Hao Chen and Ying Cui and Dongyan Guo},
  title = {{DiffCalib: Reformulating Monocular Camera Calibration as Diffusion-Based Dense Incident Map Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i3.32355},
}
Figure