Learned Uncertainty Calibration for Visual Inertial Localization

Stephanie Tsuei, Stefano Soatto, Paulo Tabuada, Mark B. Milam

TL;DR

Extended Kalman Filters (EKFs) are the workhorse of robotic localization, but they are notoriously "overconfident," providing uncertainty estimates (covariances) that are much smaller than the actual errors. This paper demonstrates that this miscalibration is systematic and can be corrected using small, efficient neural networks that map uncalibrated estimates to "ground-truth" uncertainty, significantly improving the reliability of Visual Inertial Odometry (VIO) systems.

What Problem Does This Paper Address?

When a robot moves, it needs to know both where it is (state estimation) and how sure it is about that position (uncertainty estimation). The Extended Kalman Filter (EKF) provides both: a mean state and a covariance matrix. However, because EKFs rely on linearizing complex nonlinear physics and assume noise is always Gaussian (which it often isn't in vision tasks), the resulting covariance is usually wrong.

In practice, EKFs are often "overconfident," meaning the actual error is much larger than what the filter claims. This is a major safety issue; if a self-driving car thinks its position is accurate to within 5cm but it's actually off by 50cm, it might hit a curb. While previous researchers have tried to fix this using massive deep-learning models that look at raw images, these are computationally expensive and hard to integrate into traditional robotics pipelines.

Methods

The authors propose a "post-hoc" calibration approach. Instead of redesigning the filter, they learn a nonlinear mapping to correct the output of an existing EKF.

Key technical contributions include:

  1. Low-Dimensional Inputs: Unlike prior work using raw images, the authors use only the estimated state ($\hat{x}$) and the estimated covariance ($\hat{P}$) as inputs. This allows for extremely small, fast feedforward neural networks.
  2. Calibration Metric: They use the L2 divergence between the distribution of the Normalized Estimation Error Squared (NEES) and the theoretical $\chi^2$ distribution. A well-calibrated filter should have an L2 divergence of zero.
  3. Ergodicity Assumption: Since running thousands of "Monte-Carlo" simulations on a real robot is impossible, they assume ergodicity—the idea that errors over a short window of time represent the statistical population. This allows them to compute a "pseudo-ground-truth" covariance from a single real-world trajectory.
  4. Hypothesis Testing: They compared four models of increasing complexity:
  5. A simple constant scaling factor.
  6. A constant linear transformation.
  7. A neural network mapping $\hat{P}$ to a calibrated version.
  8. A state-dependent neural network mapping both $\hat{x}$ and $\hat{P}$.

Datasets

  • Simulated Linear Kalman Filter: A spring-mass damper system used as a "perfect" baseline.
  • Simulated 2D Localization: A Dubin’s car model receiving range and bearing measurements from beacons.
  • TUM Visual Inertial (VI) Dataset: A high-quality real-world benchmark featuring aggressive motions of a camera/IMU rig in various indoor environments (room1-room6). The authors used 10 sequences for training and 2 for testing.

Key Insights and Ideas

  • Miscalibration is Systematic: The most important finding is that EKF errors aren't random; they follow a learnable pattern. This means we don't need to "fix" the EKF's internal math if we can just learn how it tends to fail.
  • Covariance Carries the Signal: Interestingly, the state-dependent model (Hypothesis 4) only provided a "modest benefit" over the model that only looked at the covariance (Hypothesis 3). This suggests that the estimated covariance $\hat{P}$ itself contains almost all the information needed to diagnose its own miscalibration.
  • Ergodicity is a Valid Proxy: The authors proved that using a temporal window to estimate ground-truth covariance (the ergodic assumption) is a viable alternative to expensive Monte-Carlo trials for real-world robotics data.

Key Results

The neural network approaches outperformed traditional scaling methods across the board:

  • 2D Simulation: The neural network reduced the divergence from the ground truth by 75.4%, while the state-dependent version reached 90.9%.
  • Real-World VIO (TUM VI): The neural network achieved a 97.8% reduction in divergence.
  • $\sigma$-Bound Improvements: In the unadjusted VIO system, only ~11% of errors fell within the 1-$\sigma$ bound (it should be ~68%). After neural calibration, this jumped to ~70-76%, aligning closely with theoretical Gaussian expectations.

Limitations and Future Work

  • Post-hoc vs. In-loop: This study adjusted the covariance after the filter finished. The authors note that using a neural network to modify the EKF during operation (in-the-loop) could create "closed-loop dependencies" that might affect the stability of the filter.
  • Generalization: The models were trained and tested on the same sensor suite and environment. It remains to be seen if a network trained on one robot/camera setup would work on a different one without retraining.
  • Recurrent Models: While feedforward networks worked well, the authors suggest that Recurrent Neural Networks (RNNs) might capture even more temporal context for better calibration.

Summary updated

Abstract

The widely-used Extended Kalman Filter (EKF) provides a straightforward recipe to estimate the mean and covariance of the state given all past measurements in a causal and recursive fashion. For a wide variety of applications, the EKF is known to produce accurate estimates of the mean and typically inaccurate estimates of the covariance. For applications in visual inertial localization, we show that inaccuracies in the covariance estimates are systematic, i.e. it is possible to learn a nonlinear map from the empirical ground truth to the estimated one. This is demonstrated on both a standard EKF in simulation and a Visual Inertial Odometry system on real-world data.
BibTeX
@inproceedings{tsuei2021learned,
  author = {Stephanie Tsuei and Stefano Soatto and Paulo Tabuada and Mark B. Milam},
  title = {{Learned Uncertainty Calibration for Visual Inertial Localization}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=u_9OwHD7Vz},
}

Loading related content...

Figure