TL;DR
Visual-inertial systems need continuous calibration to remain accurate over their lifespan, but running full-batch optimization on mobile devices drains computational resources. This paper introduces a method to automatically select only the most "informative" motion segments using an efficient entropy-based metric. By calibrating only on this sparse subset of data, the system achieves full-batch accuracy while cutting computational time by a factor of 10.
What Problem Does This Paper Address?
Visual-inertial tracking systems—which fuse data from cameras and Inertial Measurement Units (IMUs)—are the backbone of modern mobile robotics, AR/VR headsets (like Microsoft HoloLens), and smartphones. For these systems to estimate their position accurately, the sensors must be precisely calibrated. This includes intrinsic parameters (like camera focal length and IMU axis misalignment) and extrinsic parameters (the physical 3D transformation between the camera and the IMU).
While devices are calibrated in the factory, these parameters drift over time due to temperature changes, physical shocks, and general wear. Self-calibration "in the wild" is possible, but it requires specific types of dynamic motion to make all parameters observable. The standard solution is to run a "full-batch" calibration over all collected trajectory data. However, this is computationally prohibitive for battery-powered, resource-constrained consumer devices. The authors tackle the challenge of achieving full-batch calibration accuracy using only a fraction of the computational budget.
Methods
The authors propose a background self-calibration module that runs alongside a standard visual-inertial SLAM (Simultaneous Localization and Mapping) system. Instead of hoarding all trajectory data, the system chops the data stream into short "motion segments."
The core novelty lies in how the system decides which segments to keep:
- Information-Theoretic Scoring: The system evaluates the information content of each candidate segment using differential entropy. To make this mathematically tractable on mobile processors, they use a rank-revealing QR decomposition to approximate the marginal covariance of the calibration parameters, entirely avoiding expensive matrix inversions.
- Parameter Grouping: Different motions calibrate different sensors (e.g., dynamic acceleration is needed for the IMU, while simple rotation might suffice for the camera). To prevent strong signals from overshadowing weak ones, the authors split the parameters into three groups (IMU intrinsics, camera intrinsics, and extrinsics).
- Database and Sparsified Optimization: The system maintains a small database of the top $N$ most informative segments for each parameter group. Once enough data is collected, a sparsified non-linear least squares optimization is run only on this curated database.
Datasets
The authors evaluated their approach on 15 real-world datasets collected using a Google Tango Development Kit tablet, which is equipped with a MEMS IMU and a global shutter fisheye camera. The datasets consist of 3-minute hand-held trajectories in a standard room (8x6x4 meters). The trajectories feature a mix of calm motions and highly dynamic movements designed to excite all rotational and translational degrees of freedom.
Key Insights and Ideas
- Quality over Quantity: You don't need hours of data to calibrate a sensor; you just need a few seconds of the right kind of data. Sparsifying the problem doesn't degrade accuracy if your selection metric is mathematically sound.
- Parameter Decoupling is Crucial: Evaluating segment information against all parameters at once can cause well-observable parameters to drown out harder-to-observe ones. Grouping parameters and maintaining separate "top N" lists ensures all sensors get the specific motion data they need to calibrate properly.
- Strategic Approximations Yield Massive Speedups: By ignoring cross-terms between different segments when calculating entropy, the system sacrifices a tiny bit of theoretical optimality. However, this approximation makes the segment-selection process incredibly fast, easily outweighing the cost of occasionally saving a redundant segment.
Key Results
- Massive Speedup: The proposed method completed calibration in an average of 31.2 seconds. This is roughly 6x faster than the full-batch approach (178.5s) and over 10x faster than a comparable state-of-the-art segment-based method (395.9s).
- Maintained Accuracy: Despite throwing away the vast majority of the trajectory data, the estimated calibration parameters showed insignificant deviation from the computationally heavy full-batch baseline.
- Validation of the Metric: In an ablation study, the authors intentionally selected the least informative segments. This resulted in higher estimation errors and variance, proving that the entropy metric successfully identifies high-value calibration data.
Limitations and Future Work
The authors acknowledge a few limitations in their approach. First, the entropy metric is a scalar value; it summarizes information but loses directional data regarding the parameters. Second, to keep calculations fast during the selection phase, the metric ignores information coming from loop closures or cross-terms between different segments (though loop closures are used in the final optimization step).
For future work, the authors note that their grouping of calibration parameters (camera vs. IMU) was based on intuition. They suggest formally analyzing the co-observability structure of the parameters to group them mathematically, which could lead to even more optimal segment selection.
Summary updated