MATE: Machine Learning for Adaptive Calibration Template Detection

Simon Donné, Jonas De Vylder, Bart Goossens, Wilfried Philips

TL;DR

This paper introduces MATE, a novel deep learning approach using Convolutional Neural Networks (CNNs) for robust checkerboard corner detection, a crucial step in camera calibration. MATE achieves state-of-the-art accuracy while being adaptable to various distortions and noise levels, outperforming existing methods in challenging scenarios.

What Problem Does This Paper Address?

Camera calibration is fundamental for many computer vision tasks, but it relies on accurately identifying specific features (like checkerboard corners) in images. Traditional methods for finding these corners can be computationally expensive and struggle with real-world image imperfections such as lens distortion, noise, and perspective changes. This paper tackles the challenge of reliably detecting these critical calibration points, especially in the presence of significant image degradation, which is a common issue in practical applications.

Methods

The core of MATE is a three-layer Convolutional Neural Network (CNN).

  • Layer 1: This layer uses a relatively large kernel (13x13 pixels) with 16 output channels. Its purpose is to extract initial features from pixel neighborhoods, designed to be robust to image blur. It employs ReLU activation.
  • Layer 2: This layer performs a local translation of the 16 features from the first layer into 8 new features. It uses ReLU activation and combines local neighborhood characteristics without further spatial filtering, aiming to create more meaningful higher-level features.
  • Layer 3: The final layer combines the 8 features from the second layer into a single-channel response map. It uses a smaller kernel and aims to produce a final score indicating the likelihood of a checkerboard corner at each pixel.

The network is trained using stochastic gradient descent (SGD) with a momentum term to improve convergence. A key aspect is its adaptability: it can be trained on general datasets or fine-tuned for specific scenarios, such as different calibration patterns or specific types of distortions. The training objective is a one-sided quadratic difference cost function, penalizing both missed corners and false detections, with weights adjusted for the imbalance between positive and negative samples.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Training Datasets:
    • Pclean: A set of 85 captured images, augmented with rotations and intensity inversions.
  • Pfull: Pclean further augmented with artificial radial and tangential distortion, and Gaussian noise to simulate poor image quality.
  • Evaluation Datasets:
    • uEye dataset: Captured by two cameras in a wide-baseline setup, used to evaluate robustness against perspective transforms.
  • GoPro dataset: Captured with a wide-angle lens, used to evaluate robustness against significant lens distortions.
  • "Angle" dataset: Used to specifically test robustness to checkerboard skew.
  • Datasets from Placht et al. [14] (uEye and GoPro) were also used for comparison.

Key Insights and Ideas

  • Generalization through Deep Learning: The paper demonstrates that a relatively shallow CNN (three layers) can effectively learn to detect checkerboard corners, generalizing well beyond the specific training examples.
  • Adaptability is Key: The ability to retrain or fine-tune the network for specific scenarios (e.g., different calibration patterns, noise levels) is a major advantage over hand-designed feature extractors.
  • Robustness to Degradation: MATE shows remarkable resilience to lens distortion and noise, outperforming traditional methods in challenging, real-world conditions.
  • Trade-off Management: The network's design and training allow for a controllable trade-off between detecting more corners (recall) and minimizing false positives, managed through the final threshold.
  • CMYK Hexboard Illustration: The paper showcases MATE's adaptability by training it on a non-standard CMYK hexboard, demonstrating its potential for diverse calibration targets.

Key Results

  • Accuracy: MATE achieves comparable or superior accuracy to state-of-the-art methods like ChESS, ROCHADE, and OCamCalib across various datasets. For instance, on the uEye dataset, MATE achieved an accuracy of 1.009 px, comparable to CHESS (0.946 px) and better than ROCHADE (1.510 px).
  • Robustness: MATE* (trained on Pclean) and MATE (trained on Pfull) show good performance on the uEye dataset, indicating robustness to perspective transforms. MATE, trained on the distorted Pfull dataset, performs well on the GoPro dataset, demonstrating its resilience to lens distortion.
  • Speed: MATE offers improved execution times compared to some methods, particularly ROCHADE, which is noted for its slow processing of large images. MATE's processing time is generally competitive.
  • False Positives vs. Recall: MATE trained on the full dataset (Pfull) sacrifices some precision (higher false positives) for better recall (fewer missed corners), a trade-off that checkerboard construction algorithms can often handle.
  • Angle Robustness: MATE demonstrates superior performance in detecting checkerboards at extreme angles (up to 70 degrees) compared to other methods, especially when trained on distorted data.

Limitations and Future Work

  • Training Data Dependency: While adaptable, the performance of MATE is still dependent on the quality and representativeness of the training data. The paper notes that MATE* was "woefully unprepared" for the noise in the GoPro dataset, highlighting the importance of comprehensive training.
  • Sub-pixel Refinement: The evaluation focuses on raw detections. While the paper mentions that sub-pixel refinement can be applied afterwards, the core MATE method's performance is based on pixel-level accuracy.
  • Future Work: The authors suggest that the network architecture and training process can be easily adapted for new scenarios and applications, implying further exploration of different calibration targets and more complex image conditions. The impact of spatial support radius in the first layer is explored, suggesting that optimal choices depend on the application's processing time constraints and desired performance trade-offs.

Summary updated

Abstract

The problem of camera calibration is two-fold. On the one hand, the parameters are estimated from known correspondences between the captured image and the real world. On the other, these correspondences themselves—typically in the form of chessboard corners—need to be found. Many distinct approaches for this feature template extraction are available, often of large computational and/or implementational complexity. We exploit the generalized nature of deep learning networks to detect checkerboard corners: our proposed method is a convolutional neural network (CNN) trained on a large set of example chessboard images, which generalizes several existing solutions. The network is trained explicitly against noisy inputs, as well as inputs with large degrees of lens distortion. The trained network that we evaluate is as accurate as existing techniques while offering improved execution time and increased adaptability to specific situations with little effort. The proposed method is not only robust against the types of degradation present in the training set (lens distortions, and large amounts of sensor noise), but also to perspective deformations, e.g., resulting from multi-camera set-ups.
BibTeX
@inproceedings{donne2016mate,
  author = {Simon Donné and Jonas De Vylder and Bart Goossens and Wilfried Philips},
  title = {{MATE: Machine Learning for Adaptive Calibration Template Detection}},
  booktitle = {Sensors},
  year = {2016},
  url = {https://openreview.net/forum?id=kf4vqLul70},
}
Figure