Online Multi Camera-IMU Calibration

Jacob Hartzer, Srikanth Saripalli

TL;DR

This paper presents an enhanced Kalman filter framework for simultaneously calibrating the spatial and temporal relationships between multiple cameras and an Inertial Measurement Unit (IMU). It addresses the challenge of maintaining accurate sensor alignment in dynamic environments, offering a more robust and flexible solution than traditional offline methods.

What Problem Does This Paper Address?

Accurate localization for mobile robots is crucial for navigation and control, especially in environments where GPS is unreliable (indoors, urban canyons). Visual-Inertial Navigation Systems (V-INS) leverage cameras and IMUs for this, but their performance hinges on precise calibration of the sensors' relative positions, orientations, and timing. Traditional calibration methods are often offline, time-consuming, and struggle with dynamic changes or sensor shifts due to maintenance or vibration. Existing online methods, while better, often focus on single cameras and can be complex to implement, especially with multiple sensors having different data rates and synchronization capabilities. This paper tackles the need for a unified, online calibration system that can handle multiple cameras with varying characteristics and temporal inconsistencies.

Methods

The core of this work is an extension of an Extended Kalman Filter (EKF) framework for IMU-camera calibration. Key innovations include:

  • Multi-Camera Integration: The framework is generalized to incorporate multiple camera sensors, vectorizing their extrinsic parameters into a single state vector.
  • Fiducial Detection Reformulation: The measurement model is adapted to leverage readily available data from fiducial detection software (like ArUco markers), simplifying implementation and reducing developmental complexity. This allows for the use of various robust 3D sensor models.
  • One-Way Time Translation Filter: A novel, independent filter is introduced to estimate the time translation between sensors that lack closed-loop synchronization (e.g., due to hardware limitations or differing update rates). This filter uses a simple linear model and is designed to be decoupled from the main calibration filter for flexibility and to avoid increasing its complexity.
  • Error-State Kalman Filter: The propagation of the state and covariance uses an error-state formulation, which is standard for robust Kalman filtering, especially when dealing with non-linear dynamics and potential singularities.

Datasets

The paper does not explicitly mention specific public datasets used for training or evaluation. Instead, it describes experimental setups using:

  • A multi-camera IMU rig with cameras (PoE and USB 3.0) and an IMU (VectorNav).
  • A stationary ChArUco calibration target.
  • Experimental data collected by the authors.

The authors also compare their results against the MatLab camera calibration toolbox, which can be considered a benchmark for extrinsic calibration.

Key Insights and Ideas

  • Decoupled Time Synchronization is Powerful: The introduction of a separate, one-way time translation filter significantly simplifies the calibration process for sensors with inconsistent timing. This approach is more flexible than integrating timing directly into the main EKF state, especially when dealing with many sensors or varying synchronization methods.
  • Fiducial Detection Simplifies Integration: By reformulating the measurement model to work with standard fiducial detection outputs, the framework becomes highly adaptable. This means it can quickly incorporate new camera sensors or different types of fiducial markers without extensive re-engineering.
  • Non-Overlapping Fields of View are Not a Showstopper: A key finding is that the EKF can still converge even when cameras do not have overlapping fields of view. This is because the IMU acts as a unifying sensor, allowing cameras to contribute to reducing uncertainty even when they aren't simultaneously observing the calibration target.
  • Extensibility is a Major Advantage: The vectorized approach to camera extrinsics makes the framework inherently extensible to any number of cameras, a significant practical benefit for complex robotic systems.

Key Results

  • Fast Convergence: The time translation filter converged quickly (within seconds) for both IMU and camera sensors, significantly reducing timing noise.
  • Accurate Extrinsic Calibration: The multi-camera EKF demonstrated consistent convergence for extrinsic parameters, with results comparable to the MatLab camera calibration toolbox (relative differences in physical location were low, typically within a few percent).
  • Robustness to Non-Overlapping Views: Experiments showed that cameras facing different directions could still achieve convergent calibrations, with measurements from one camera improving the uncertainty of another indirectly through the IMU.
  • Reduced Covariance: Even cameras not directly observing the calibration target benefited from reduced uncertainty due to measurements from other cameras in the system.

Limitations and Future Work

  • Assumptions on Fiducials: The framework relies on fiducial markers being robust 3D sensors with low error in position and orientation.
  • Limited Excitation: In some scenarios, the calibration target's size or placement might limit the excitation in certain degrees of freedom, leading to higher error in those specific parameters (e.g., the Y-offset in the experimental results).
  • Future Work: The authors suggest further exploration of temporal calibration and its impact on sensor fusion. They also highlight the extensibility of their framework for future applications.

Summary updated

Abstract

Visual-inertial navigation systems are powerful in their ability to accurately estimate localization of mobile systems within complex environments that preclude the use of global navigation satellite systems. However, these navigation systems are reliant on accurate and up-to-date temporospatial calibrations of the sensors being used. As such, online estimators for these parameters are useful in resilient systems. This paper presents an extension to existing Kalman Filter based frameworks for estimating and calibrating the extrinsic parameters of multi-camera IMU systems. In addition to extending the filter framework to include multiple camera sensors, the measurement model was reformulated to make use of measurement data that is typically made available in fiducial detection software. A secondary filter layer was used to estimate time translation parameters without closed-loop feedback of sensor data. Experimental calibration results, including the use of cameras with non-overlapping fields of view, were used to validate the stability and accuracy of the filter formulation when compared to offline methods. Finally the generalized filter code has been open-sourced and is available online.
BibTeX
@inproceedings{hartzer2022online,
  author = {Jacob Hartzer and Srikanth Saripalli},
  title = {{Online Multi Camera-IMU Calibration}},
  booktitle = {Ssrr},
  year = {2022},
  doi = {10.1109/ssrr56537.2022.10018692},
}
Figure