TL;DR
This paper introduces MATE, a novel deep learning approach using Convolutional Neural Networks (CNNs) for robust checkerboard corner detection, a crucial step in camera calibration. MATE achieves state-of-the-art accuracy while being adaptable to various distortions and noise levels, outperforming existing methods in challenging scenarios.
What Problem Does This Paper Address?
Camera calibration is fundamental for many computer vision tasks, but it relies on accurately identifying specific features (like checkerboard corners) in images. Traditional methods for finding these corners can be computationally expensive and struggle with real-world image imperfections such as lens distortion, noise, and perspective changes. This paper tackles the challenge of reliably detecting these critical calibration points, especially in the presence of significant image degradation, which is a common issue in practical applications.
Methods
The core of MATE is a three-layer Convolutional Neural Network (CNN).
- Layer 1: This layer uses a relatively large kernel (13x13 pixels) with 16 output channels. Its purpose is to extract initial features from pixel neighborhoods, designed to be robust to image blur. It employs ReLU activation.
- Layer 2: This layer performs a local translation of the 16 features from the first layer into 8 new features. It uses ReLU activation and combines local neighborhood characteristics without further spatial filtering, aiming to create more meaningful higher-level features.
- Layer 3: The final layer combines the 8 features from the second layer into a single-channel response map. It uses a smaller kernel and aims to produce a final score indicating the likelihood of a checkerboard corner at each pixel.
The network is trained using stochastic gradient descent (SGD) with a momentum term to improve convergence. A key aspect is its adaptability: it can be trained on general datasets or fine-tuned for specific scenarios, such as different calibration patterns or specific types of distortions. The training objective is a one-sided quadratic difference cost function, penalizing both missed corners and false detections, with weights adjusted for the imbalance between positive and negative samples.
Datasets
The paper utilizes several datasets for training and evaluation:
- Training Datasets:
Pclean: A set of 85 captured images, augmented with rotations and intensity inversions.
Pfull:Pcleanfurther augmented with artificial radial and tangential distortion, and Gaussian noise to simulate poor image quality.- Evaluation Datasets:
uEyedataset: Captured by two cameras in a wide-baseline setup, used to evaluate robustness against perspective transforms.
GoProdataset: Captured with a wide-angle lens, used to evaluate robustness against significant lens distortions.- "Angle" dataset: Used to specifically test robustness to checkerboard skew.
- Datasets from Placht et al. [14] (uEye and GoPro) were also used for comparison.
Key Insights and Ideas
- Generalization through Deep Learning: The paper demonstrates that a relatively shallow CNN (three layers) can effectively learn to detect checkerboard corners, generalizing well beyond the specific training examples.
- Adaptability is Key: The ability to retrain or fine-tune the network for specific scenarios (e.g., different calibration patterns, noise levels) is a major advantage over hand-designed feature extractors.
- Robustness to Degradation: MATE shows remarkable resilience to lens distortion and noise, outperforming traditional methods in challenging, real-world conditions.
- Trade-off Management: The network's design and training allow for a controllable trade-off between detecting more corners (recall) and minimizing false positives, managed through the final threshold.
- CMYK Hexboard Illustration: The paper showcases MATE's adaptability by training it on a non-standard CMYK hexboard, demonstrating its potential for diverse calibration targets.
Key Results
- Accuracy: MATE achieves comparable or superior accuracy to state-of-the-art methods like ChESS, ROCHADE, and OCamCalib across various datasets. For instance, on the
uEyedataset, MATE achieved an accuracy of 1.009 px, comparable to CHESS (0.946 px) and better than ROCHADE (1.510 px). - Robustness: MATE* (trained on
Pclean) and MATE (trained onPfull) show good performance on theuEyedataset, indicating robustness to perspective transforms. MATE, trained on the distortedPfulldataset, performs well on theGoProdataset, demonstrating its resilience to lens distortion. - Speed: MATE offers improved execution times compared to some methods, particularly ROCHADE, which is noted for its slow processing of large images. MATE's processing time is generally competitive.
- False Positives vs. Recall: MATE trained on the full dataset (
Pfull) sacrifices some precision (higher false positives) for better recall (fewer missed corners), a trade-off that checkerboard construction algorithms can often handle. - Angle Robustness: MATE demonstrates superior performance in detecting checkerboards at extreme angles (up to 70 degrees) compared to other methods, especially when trained on distorted data.
Limitations and Future Work
- Training Data Dependency: While adaptable, the performance of MATE is still dependent on the quality and representativeness of the training data. The paper notes that MATE* was "woefully unprepared" for the noise in the
GoProdataset, highlighting the importance of comprehensive training. - Sub-pixel Refinement: The evaluation focuses on raw detections. While the paper mentions that sub-pixel refinement can be applied afterwards, the core MATE method's performance is based on pixel-level accuracy.
- Future Work: The authors suggest that the network architecture and training process can be easily adapted for new scenarios and applications, implying further exploration of different calibration targets and more complex image conditions. The impact of spatial support radius in the first layer is explored, suggesting that optimal choices depend on the application's processing time constraints and desired performance trade-offs.
Summary updated