TL;DR
This paper tackles the challenge of robustly fusing visual and inertial sensor data for 3D motion estimation, where visual data is notoriously noisy and prone to outliers. They derive an optimal, but computationally expensive, method for distinguishing good (inlier) from bad (outlier) visual measurements and propose practical approximations that significantly improve state-of-the-art visual-inertial systems without increasing computational cost.
What Problem Does This Paper Address?
Estimating a robot's or sensor platform's 3D motion (ego-motion) is crucial for navigation and mapping. A common approach is to fuse data from an Inertial Measurement Unit (IMU) and a camera. While IMUs provide continuous motion estimates, they drift over time. Cameras, on the other hand, can provide absolute pose information by tracking features in the environment. However, visual tracking is highly susceptible to errors: a significant portion (60-90%) of tracked features can become "outliers" due to factors like illumination changes, occlusions, or independently moving objects. These outliers can severely degrade the accuracy of motion estimation algorithms. Existing visual-inertial navigation systems (VINS) often employ some form of outlier rejection, but their methods are diverse and not analytically compared to an optimal approach. This paper aims to bridge this gap by deriving the theoretically optimal way to handle outliers in visual-inertial fusion and proposing practical, efficient approximations.
Methods
The core of the paper lies in deriving a robust inference framework. They start by formulating the visual-inertial system as a state-space model, incorporating IMU measurements (gyroscope and accelerometer) and visual feature observations. A key challenge is that visual measurements are often corrupted by outliers.
The authors derive an "optimal discriminant" for classifying measurements as inliers or outliers. This optimal method, based on standard hypothesis testing, is computationally intractable due to the need to consider all possible combinations of inliers and outliers. To overcome this, they propose several approximations:
- Smoother-based approach: Instead of a standard filter, they use a smoother, which leverages a "delay-line" in the state to consider past measurements, improving the ability to make informed decisions about current outliers.
- Approximations to the optimal discriminant: They explore various ways to simplify the computation of the inlier/outlier decision. This includes:
- Zero-point RANSAC: A bootstrapping approach that starts with an empty inlier set and iteratively adds features.
- One-point RANSAC: A simplified version that uses a minimal set of features to estimate motion and test others.
- History-of-innovation test: This is a novel contribution where they analyze the "whiteness" (statistical properties) of the innovation (the difference between predicted and actual measurements) over time. If the innovation deviates from expected white noise behavior, it suggests an outlier. This approach is shown to be particularly effective.
- Handling nuisance parameters: They address how to deal with unknown parameters associated with visual features (like their 3D position), proposing methods to either include them in the state or marginalize them out.
Datasets
The paper evaluates their methods on real-world data collected from a hand-held platform in various challenging environments, including:
- Specular hallways with handheld motion.
- Aggressive motion loops.
- Forested areas with occlusions.
- Crowded halls with independently moving objects.
They do not introduce new datasets but use their own collected data to compare their proposed methods against variations of existing robust filtering techniques.
Key Insights and Ideas
- The "optimal" outlier detection is computationally prohibitive: The theoretically best way to decide if a visual measurement is an outlier requires considering all possible subsets of inliers, which is infeasible.
- Leveraging the history of innovations is crucial: Analyzing the statistical properties (whiteness) of the difference between predicted and actual measurements over time provides a powerful signal for detecting outliers, even when instantaneous measurements are ambiguous.
- Smoother-based approaches offer advantages: Using a smoother, which considers past measurements, allows for more informed outlier rejection compared to standard filters.
- Practical approximations can match theoretical gains: The proposed approximations, particularly those leveraging the innovation history and one-point RANSAC, achieve performance close to the theoretical optimum without significant computational overhead.
- Robustness is key for real-world VINS: Even state-of-the-art VINS systems can be significantly degraded by outliers, highlighting the importance of robust inference techniques.
Key Results
The paper demonstrates that their proposed robust inference scheme, particularly the combination of a smoother with a history-of-innovation test and one-point RANSAC, significantly improves the performance of state-of-the-art visual-inertial sensor fusion systems.
- They show that their best-performing method (m4 in their experiments, which uses 1-point RANSAC with history-of-innovation testing) achieves the lowest trajectory error (measured by dynamic time-warping distance, $w_d$) across various challenging scenarios.
- For instance, in a ~180m loop through a natural forested area with occlusions, their method achieved a $w_d$ of 4.36, significantly outperforming other tested methods (e.g., m1 with $w_d$ of 6.53, m2 with $w_d$ of 3.68).
- Crucially, these performance gains are achieved without increasing the computational complexity, running at frame-rate on a standard processor.
Limitations and Future Work
- Computational complexity of the optimal discriminant: The paper acknowledges that the derived optimal discriminant is intractable and focuses on approximations.
- Approximation trade-offs: While the proposed approximations are effective, they are still approximations. The paper discusses the conditions under which these approximations are valid.
- Initialization: The paper mentions that "loose integration" systems (where sensor modalities are fused post-mortem) are delicate in the absence of inertial measurements for initialization. While their approach is more integrated, robust initialization for VINS remains a general challenge.
- Future work: The paper implicitly suggests further exploration of the trade-offs between filtering and smoothing, and the optimal way to incorporate nuisance parameters. The experimental validation section also hints at the need for standardized benchmarks for VINS.
Summary updated