NMC3D: Non-Overlapping Multi-Camera Calibration Based on Sparse 3D Map

Changshuai Dai, Ting Han, Yang Luo, Mengyi Wang, Guorong Cai, Jinhe Su, Zheng Gong, Niansheng LiuSensors 2024 · OpenReview

TL;DR

This paper introduces NMC3D, a novel method for calibrating multi-camera systems, particularly those lacking overlapping fields of view. By leveraging sparse 3D maps generated from individual cameras and a distance-based feature point selection strategy, NMC3D achieves accurate extrinsic parameter estimation, even in challenging scenarios like 180-degree rotations.

What Problem Does This Paper Address?

Accurate calibration of multi-camera systems is crucial for applications like robotics and autonomous systems, enabling precise perception and decision-making. However, a significant challenge arises when cameras in a system do not share overlapping views. Traditional calibration methods, which rely on feature points from calibration targets within these overlapping regions, fail in such scenarios. Existing solutions often involve complex setups like plane mirrors, increasing cost and reducing portability. Furthermore, even when feature matches are found, simply selecting them randomly for calibration doesn't fully exploit their spatial distribution and potential for accurate extrinsic parameter estimation. This paper tackles these limitations by proposing a robust and practical calibration method for non-overlapping multi-camera systems using natural environments.

Methods

The proposed NMC3D method comprises three main modules:

  1. Mapping Module: This module constructs sparse 3D maps for each individual camera. It utilizes a SLAM (Simultaneous Localization and Mapping) algorithm, inspired by ORB-SLAM3, to process input images, extract features, estimate camera poses, and build 3D maps. To enhance map accuracy, a constraint component is added, utilizing known 2D distances between corner points on calibration plates within the environment to refine the 3D map points.
  2. Calibration Module: This module takes the generated 3D maps as input and calculates the initial extrinsic transformation between cameras. It first identifies similar keyframes across the maps using a bag-of-words model. Then, it performs feature point matching between these similar keyframes. Crucially, instead of random selection, it employs a distance-based feature matching point selection strategy. This involves classifying feature points into "far," "medium," and "near" distance sets and selecting points from each to ensure a uniform distribution and leverage their varying influence on rotation and translation vector computation. The extrinsic transformation is then computed using these selected points.
  3. Optimization Module: The initial extrinsic parameters obtained from the calibration module are further refined using bundle adjustment. This process projects 3D map points from one camera's map onto the image plane of another and minimizes the reprojection error between these projected points and the actual detected points. This iterative optimization refines the extrinsic transformation relationship. For systems with more than two cameras, a primary camera is designated, and auxiliary camera extrinsics are derived relative to it.

Datasets

The paper does not explicitly mention or introduce new datasets. Instead, it utilizes:

  • Virtual Environment (Gazebo): Experiments were conducted in a simulated environment with front and rear monocular cameras.
  • Real Environment: Experiments were performed using Realsense D435i depth cameras in a real-world setting.

Key Insights and Ideas

  • Leveraging Natural Environments: The method's ability to calibrate without specific calibration sites or mirrors, relying instead on natural environments, significantly reduces cost and increases practicality.
  • Distance-Based Feature Point Selection: This is a core innovation. The insight that feature points at different distances have varying impacts on estimating rotation and translation vectors is key. By strategically selecting points across these distances and ensuring uniform distribution, the method achieves more accurate and robust extrinsic parameter estimation.
  • Sparse 3D Maps for Non-Overlapping Views: The use of individual camera-generated sparse 3D maps as a common reference, even without direct visual overlap, provides a robust foundation for calibration.
  • Robustness to Rotations: The method demonstrates effectiveness even when cameras undergo significant rotations (e.g., 180 degrees), a scenario that often challenges traditional methods.

Key Results

The paper presents experimental results in both virtual and real environments, demonstrating the effectiveness of NMC3D.

  • Virtual Environment:
    • Table 1 shows a comparison with the "Cammap" algorithm, where NMC3D achieved smaller inaccuracies in both rotation and translation compared to Cammap. For instance, NMC3D had rotation inaccuracies of [0.950, 0.430, 0.415] compared to Cammap's [1.890, 0.514, 0.210].
  • Table 2 indicates improved stability for NMC3D, with a lower overall variance (0.6626) compared to Cammap (0.6993).
  • Real Environment:
    • Tables 3 and 4 compare the extrinsic parameters obtained by NMC3D with ground truth values for front-rear and front-side camera configurations. The results show good agreement, with minor deviations attributed to the inherent challenges of real-world calibration. For example, in Table 3, the rotation for front-rear cameras was [-177.399, -4.1353, -177.957] compared to the ground truth of [-180,0,-180].

Limitations and Future Work

The authors acknowledge the following limitations:

  • Intrinsic Parameter Requirement: The method requires the intrinsic parameters of each camera to be known beforehand.
  • Feature Texture Dependency: In environments with insufficient feature texture, the accuracy of feature point matching can degrade, potentially impacting calibration accuracy.
  • Same Image Capture: For calibration, it's necessary for both cameras to capture the same scene or image, which is a prerequisite for feature point matching.

Future work directions include:

  • Automated Intrinsic and Extrinsic Calibration: Developing a system for simultaneous and highly automated calibration of both intrinsic and extrinsic parameters.
  • Handling Low-Texture Environments: Improving calibration accuracy in challenging environments by incorporating line features alongside point features.

Summary updated

Abstract

With the advancement of computer vision and sensor technologies, many multi-camera systems are being developed for the control, planning, and other functionalities of unmanned systems or robots. The calibration of multi-camera systems determines the accuracy of their operation. However, calibration of multi-camera systems without overlapping parts is inaccurate. Furthermore, the potential of feature matching points and their spatial extent in calculating the extrinsic parameters of multi-camera systems has not yet been fully realized. To this end, we propose a multi-camera calibration algorithm to solve the problem of the high-precision calibration of multi-camera systems without overlapping parts. The calibration of multi-camera systems is simplified to the problem of solving the transformation relationship of extrinsic parameters using a map constructed by multiple cameras. Firstly, the calibration environment map is constructed by running the SLAM algorithm separately for each camera in the multi-camera system in closed-loop motion. Secondly, uniformly distributed matching points are selected among the similar feature points between the maps. Then, these matching points are used to solve the transformation relationship between the multi-camera external parameters. Finally, the reprojection error is minimized to optimize the extrinsic parameter transformation relationship. We conduct comprehensive experiments in multiple scenarios and provide results of the extrinsic parameters for multiple cameras. The results demonstrate that the proposed method accurately calibrates the extrinsic parameters for multiple cameras, even under conditions where the main camera and auxiliary cameras rotate 180°.
BibTeX
@inproceedings{dai2024nmc,
  author = {Changshuai Dai and Ting Han and Yang Luo and Mengyi Wang and Guorong Cai and Jinhe Su and Zheng Gong and Niansheng Liu},
  title = {{NMC3D: Non-Overlapping Multi-Camera Calibration Based on Sparse 3D Map}},
  booktitle = {Sensors},
  year = {2024},
  url = {https://openreview.net/forum?id=RBe80aUwJQ},
}
Figure