Geometric Calibration for Cameras with Inconsistent Imaging Capabilities

Ke Wang, Chuhao Liu, Shaojie Shen

TL;DR

This paper introduces a novel camera calibration method that accounts for inconsistent imaging quality across a camera's sensor. By measuring and incorporating pixel-level uncertainties into the calibration process, their method significantly improves accuracy, especially for cameras with uneven performance, outperforming existing techniques.

What Problem Does This Paper Address?

Traditional camera calibration methods, like those in OpenCV or MATLAB, often assume that a camera's imaging quality is uniform across its entire sensor. This assumption breaks down when cameras have manufacturing defects, lens issues (like field curvature), or physical damage, leading to some areas of the image being sharper than others. These "inconsistent imaging capabilities" introduce varying levels of uncertainty in the detected calibration points (e.g., corners of a chessboard). Standard calibration algorithms, which typically use Maximum Likelihood Estimation (MLE) and assume uniform uncertainty, can be misled by these variations. They tend to "overfit" to the less reliable points in the blurry regions, resulting in inaccurate overall calibration parameters. While factors like camera pose and chessboard slant also introduce uncertainty, these can often be mitigated with sufficient data sampling. However, the uncertainty arising from inconsistent imaging quality is persistent and requires a more sophisticated approach.

Methods

The core of this paper's contribution lies in its method for measuring and utilizing pixel-level uncertainties.

  1. Uncertainty Measurement: Instead of assuming uniform uncertainty, the authors propose a method to estimate the uncertainty of each detected control point. This involves:
  2. Calculating the autocorrelation matrix of a local window around each detected point.
  3. Addressing the multi-mapping problem (where eigenvalues of the autocorrelation matrix don't uniquely determine uncertainty) by introducing a self-defined Signal-to-Noise Ratio (SNR) and using normalizing operations.
  4. Developing a predictive model (Equation 6) that relates a normalized point response to a normalized standard deviation (SD) of the control point, which is shown to be stable across various imaging conditions (open angles, SNRs, rotation angles).
  5. Using this relationship to estimate the inverse covariance matrix for each control point (Equation 7).
  6. Incorporating Uncertainties into Optimization: The standard camera calibration optimization problem (minimizing reprojection error) is modified. Instead of a simple sum of squared errors, the new objective function (Equation 4) incorporates the estimated inverse covariance matrix for each control point. This effectively weights points with smaller uncertainties (sharper regions) more heavily and down-weights points with larger uncertainties (blurrier regions) in the final optimization.
  7. Iterative Refinement: The method also includes an iterative refinement step for control point localization, similar to existing work, to further reduce the impact of lens distortion.

Datasets

The authors created a custom calibration dataset using eight different imaging systems. These systems comprised four high-resolution and four low-resolution imaging setups. They specifically selected systems with varying degrees of "inconsistent imaging capabilities," ranging from "very inconsistent" to "good." For each system, they generated 50 training and testing sequences, resulting in a total of 400 sequences.

Key Insights and Ideas

  • Uncertainty is Not Uniform: The fundamental insight is that camera sensors are not perfect and exhibit varying image quality, leading to non-uniform uncertainty in detected features. This is a critical factor often overlooked by traditional methods.
  • Autocorrelation as a Proxy for Uncertainty: The autocorrelation matrix of a local image patch around a feature point provides a useful, though initially complex, indicator of its uncertainty.
  • SNR and Normalization are Key: The authors demonstrate that by carefully defining an SNR and applying normalization techniques, the complex relationship between image properties and point uncertainty can be simplified into a predictable, single-mapping function.
  • Weighted Optimization is Crucial: Directly incorporating these measured, point-specific uncertainties into the calibration optimization (e.g., using inverse covariance matrices) is essential for achieving accurate results, especially for cameras with significant imaging inconsistencies. This weighting scheme allows the algorithm to "trust" the more reliable points more.

Key Results

The paper demonstrates the effectiveness of their method through quantitative comparisons:

  • Superior Accuracy: Their proposed method ("Ours") consistently achieved the lowest average projection error across all tested camera systems, outperforming two recent methods ([3,4]) and the standard OpenCV calibration module.
  • Performance on Inconsistent Systems: The improvements were most pronounced for cameras with "Poor" consistency, where their method significantly reduced the average reprojection error compared to others. For "Good" consistency systems, their performance was comparable to the best competing methods.
  • Statistical Performance: Figure 10 shows that their method accumulated the highest number of images with projection errors below a given threshold, indicating better overall calibration performance.
  • Runtime: The method's runtime analysis indicates that the eigenvalue decomposition for uncertainty estimation adds minimal computational cost, making it practical for real-world applications.

Limitations and Future Work

  • Image Noise Calibration: The authors note that accurate image noise calibration is essential for their method. Inaccurate noise calibration can lead to incomparable pixel uncertainties.
  • Motion Blur: In future work, they plan to investigate the negative influence of motion blur on geometric calibration and explore methods to achieve good calibration accuracy from sequences with significant motion blur.

Summary updated

Abstract

Traditional calibration methods rely on the accurate localization of the chessboard points in images and their maximum likelihood estimation (MLE)-based optimization models implicitly require all detected points to have an identical uncertainty. The uncertainties of the detected control points are mainly determined by camera pose, the slant of the chessboard and the inconsistent imaging capabilities of the camera. The negative influence of the uncertainties that are induced by the two former factors can be eliminated by adequate data sampling. However, the last factor leads to the detected control points from some sensor areas having larger uncertainties than those from other sensor areas. This causes the final calibrated parameters to overfit the control points that are located at the poorer sensor areas. In this paper, we present a method for measuring the uncertainties of the detected control points and incorporating these measured uncertainties into the optimization model of the geometric calibration. The new model suppresses the influence from the control points with large uncertainties while amplifying the contributions from points with small uncertainties for the final convergence. We demonstrate the usability of the proposed method by first using eight cameras to collect a calibration dataset and then comparing our method to other recent works and the calibration module in OpenCV using that dataset.
BibTeX
@inproceedings{wang2022geometric,
  author = {Ke Wang and Chuhao Liu and Shaojie Shen},
  title = {{Geometric Calibration for Cameras with Inconsistent Imaging Capabilities}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=_RQ7hPobqbu},
}
Figure