Abstract
Recent advancements in autonomous driving rely heavily on the precise extrinsic calibration of LiDAR and camera sensors. While deep learning models promise "target-less" online calibration by regressing transformation parameters directly from data, their real-world generalization remains questionable. This paper investigates the generalization capabilities of these models, revealing that most "regression-based" architectures function as retrieval networks that memorize spatial distributions rather than learning to match features. Through systematic analysis and cross-camera experiments, we demonstrate that explicit geometric matching is essential for robust, generalizable calibration.
Introduction: The Calibration Challenge
For autonomous vehicles to perceive their environment accurately, data from cameras (providing color and texture) and LiDAR (providing 3D geometry) must be perfectly aligned. This alignment, known as extrinsic calibration, traditionally requires manual effort and specific targets like checkerboards.
Learning-based methods aim to automate this process. However, many models that perform well on specific benchmarks fail when deployed in different sensor configurations. This study investigates whether these networks are truly learning geometric relationships or simply "cheating" by memorizing the training dataset's spatial characteristics.
Methodological Paradigms
The authors categorize existing learning-based calibration into two main paradigms:
- Regression-based methods: These networks directly output extrinsic parameters (rotation and translation) from an input image and point cloud.
- Matching-based methods: These networks identify correspondences between 2D pixels and 3D points, subsequently employing a geometric solver (e.g., PnP) to determine the calibration.
To test generalization, the authors conducted a "cross-camera" experiment using a stereo rig. A model trained on the left camera was tested on the right camera. Since the sensors are physically offset but otherwise identical, a model that has learned true geometric matching should generalize, whereas a model that has memorized spatial coordinates will fail.
Key Insights: The "Retrieval" Trap
The investigation revealed that regression-based networks often ignore the camera-LiDAR relationship. Instead, they learn a mapping between the specific shape of a depth map and an extrinsic value. This "retrieval" behavior is evidenced by the fact that these models can achieve high accuracy on training sets even when the RGB image is removed entirely, focusing solely on the LiDAR distribution.
Furthermore, standard data generation techniques—adding random "jitters" to perfectly calibrated frames—fail to simulate how a LiDAR's viewpoint actually changes with sensor displacement. This leads to models that are highly sensitive to camera intrinsics (like cropping or resizing) and physical sensor placement, as they lack a mechanism to account for these changes.
Results: The Smoking Gun
The quantitative results highlight a massive "generalization gap." When testing a model trained on the left camera on the right camera, translation error on the x-axis spiked significantly. For instance, LCCNet's translation error jumped from 1.31 cm to 52.48 cm.
As shown in the error distribution, the translation error for the right camera (specifically on the X-axis) is numerically close to the physical distance between the two cameras. This confirms that the regression network was simply outputting the transformation it "remembered" for the left camera, regardless of the visual input. In contrast, matching-based methods like MDPCalib proved far more robust, with error only increasing from 0.18 cm to 2.94 cm during the cross-camera swap.
Conclusion and Future Work
This research serves as a critical diagnostic for the current state of learning-based calibration. To move forward, the community should:
- Shift from pure regression toward matching-based paradigms that leverage explicit geometric constraints.
- Develop sophisticated data generation pipelines that simulate diverse sensor placements rather than simple perturbations.
- Incorporate camera intrinsic parameters directly into the learning process to handle variations in lenses and image processing.
By focusing on true geometric matching, we can develop calibration systems that are reliable across diverse real-world robotic and automotive platforms.
Summary updated