Event-IMU fusion strategies for faster-than-IMU estimation throughput

William Chamorro, Joan Solà, Juan Andrade-CettoWorkshops @ CVPR 2023 · DOI

Introduction

In high-speed robotic vision, event cameras (neuromorphic sensors) offer a significant advantage over traditional frame-based cameras due to their microsecond latency and asynchronous reporting of pixel-level intensity changes. However, a fundamental bottleneck exists in standard sensor fusion architectures: the "predict-with-IMU, correct-with-vision" approach is limited by the sampling rate of the Inertial Measurement Unit (IMU), typically 1 kHz. When a robot moves at extreme speeds, events are generated much faster than IMU readings arrive, causing the system to lose the low-latency benefits of the event camera.

This paper introduces a novel fusion paradigm that treats IMU data as a measurement rather than the primary prediction driver. By flipping this relationship, the researchers achieve a pose estimation throughput of 10 kHz—ten times faster than the IMU sampling rate and 100 times faster than current state-of-the-art event-VIO systems.

Methodology: Events-First Fusion

The proposed tracking system is built upon a Lie Kalman Filter (LKF) integrated into a Parallel Tracking and Mapping (PTAM) framework. The core innovation is the Events-First Fusion strategy, which prioritizes the high-frequency event stream for state prediction.

1. Prediction and Correction Models

Instead of using the IMU to drive the prediction step, the filter predicts motion using kinematic models at 10 kHz. IMU data is then incorporated as a correction measurement whenever it becomes available. Two primary variants were developed:

  • CV+MA Model: Constant Velocity prediction combined with Mean Acceleration measurements (averaging IMU data over the sampling window).
  • CA+IA Model: Constant Acceleration prediction combined with Instantaneous Acceleration measurements.
The proposed Events-First tracking architecture, where high-frequency event data drives prediction and IMU data provides corrections.

2. Line-Based Tracking

To ensure robustness in high-speed scenarios, the system tracks 3D lines rather than individual points. Lines provide stronger geometric constraints and are more resilient to the rapid motion dynamics encountered in man-made environments like offices or factories.

Experimental Results and Stress Testing

The system was evaluated using a DAVIS346 sensor and validated against an Optitrack motion capture system. To find the breaking point of the algorithms, the researchers conducted a "Stress Test" using a custom four-bar mechanism capable of rotating the sensor from 240 RPM up to 990 RPM.

Tracking performance during the stress test. The system maintains high accuracy across increasing rotation frequencies (up to 16.5 Hz), significantly outperforming traditional frame-based methods.

Key Insights:

  • Throughput and Accuracy: The system achieved a 10 kHz pose estimation rate with a 5% accuracy improvement over event-only estimators. It significantly outperformed frame-based visual odometry (SVO), which fails due to motion blur at these speeds.
  • Overcoming IMU Saturation: During extreme maneuvers, the physical IMU hardware often saturated at 16g. Because the system uses an events-first prediction model, it could sometimes maintain tracking even when the IMU was physically blinded by its own limits.
  • Computational Efficiency: The sensor fusion variants added only approximately 6% computational overhead compared to event-only methods, making the 10 kHz throughput feasible for real-time applications.

Conclusion

By re-architecting the relationship between event cameras and IMUs, this research demonstrates that estimation throughput can effectively "break" the IMU sampling barrier. The resulting 10 kHz tracking system provides the ultra-low latency required for the next generation of high-speed autonomous drones and robotic platforms operating in structured environments.

Summary updated

BibTeX
@inproceedings{chamorro2023event,
  author = {William Chamorro and Joan Solà and Juan Andrade-Cetto},
  title = {{Event-IMU fusion strategies for faster-than-IMU estimation throughput}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00414},
}
Figure