milliEgo: single-chip mmWave radar aided egomotion estimation via deep sensor fusion

Chris Xiaoxuan Lu, Muhamad Risqi Utama Saputra, Peijun Zhao, Yasin Almalioglu, Pedro P. B. de Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, Andrew Markham

TL;DR

milliEgo is the first deep-learning-based odometry system that uses low-cost, single-chip mmWave radar and IMU data to track a device's movement (egomotion) in 3D space. By transforming sparse, noisy radar point clouds into structured images and using a novel "mixed attention" fusion mechanism, it achieves a remarkably low 1.3% distance drift, outperforming traditional visual and inertial methods in challenging environments like dark rooms or glass corridors.

What Problem Does This Paper Address?

Accurate egomotion estimation—knowing exactly how a robot or person is moving—is the backbone of autonomous navigation and AR/VR. Currently, most systems rely on Visual-Inertial Odometry (VIO). While effective, VIO fails in the dark, in blinding light, or in "featureless" areas (like a hallway with plain white walls). Lidar is a robust alternative but remains too bulky and expensive for small robots or smartphones.

Millimeter-wave (mmWave) radar is cheap, small, and works in any lighting or weather. However, the data it produces is "messy":

  1. Extreme Sparsity: A typical mmWave scan has 100x fewer points than a Lidar scan.
  2. High Noise: Up to 75% of the detected points can be "ghost" reflections caused by multi-path interference (signals bouncing off multiple surfaces). Traditional geometric algorithms (like Iterative Closest Point) simply break down when faced with such low-quality data.

Methods

The authors propose a deep-learning pipeline that treats egomotion as a learning problem rather than a purely geometric one.

  • Panoramic Projection: To make unstructured 3D radar points digestible for a neural network, they project the points onto a 2D "depth" image using a panoramic view. This allows them to use powerful Convolutional Neural Networks (CNNs) to extract motion features.
  • Mixed Attention Fusion (The "Secret Sauce"): Instead of just concatenating radar and IMU data, they use a two-stage attention mechanism:
  • Stage 1 (Intra-sensor): Each sensor "looks at itself" to filter out its own noise (e.g., the radar ignoring multi-path reflections).
  • Stage 2 (Inter-sensor): The sensors "cross-talk" to decide which one is more reliable at any given moment. If the IMU is glitching or the radar has low visibility, the model adaptively re-weights their contributions.
  • Temporal Modeling: A two-layer LSTM (Long Short-Term Memory) network tracks the movement over time, ensuring the trajectory remains smooth and consistent.

Datasets

The researchers collected a massive, diverse dataset to prove their system's robustness:

  • Scale: Over 8km of trajectories across 17 distinct floors in 6 different commercial buildings.
  • Hardware: Data was collected using both a mobile robot (Turtlebot 2) and a custom 3D-printed handheld device.
  • Ground Truth: They used high-end, "bulky" Lidar sensors to provide the "perfect" trajectory for training and evaluation.
  • Diversity: The environments included hallways, canteens, atrium areas, and cluttered storerooms, specifically targeting areas with glass walls and dynamic lighting that confuse cameras.

Key Insights and Ideas

  • Learning beats Geometry for "Bad" Data: Traditional math-heavy registration fails on sparse radar data because there aren't enough points to "match." Deep learning can learn the underlying patterns of motion even when the data is incomplete.
  • The Power of "Cross-talk": The authors found that single-stage attention (common in other models) makes sensors change weights synchronously. Their two-stage "Mixed Attention" allows sensors to dominate interchangeably—for example, the IMU takes over during sharp turns where radar might struggle with parallax.
  • Efficiency is Possible: Despite using complex DNNs, the model is small enough (107MB) to run at 17 FPS on an NVIDIA Jetson TX2, making it viable for real-time use on embedded hardware.

Key Results

  • Accuracy: milliEgo achieved a 1.3% 3D error drift. In a typical test, it had a mean Absolute Trajectory Error (ATE) of just 0.814 meters.
  • Superiority: It significantly outperformed baseline methods, including VINET (Visual-Inertial) and DINET (Depth-Inertial), especially in environments with glass or poor lighting.
  • Robustness: In a "Dark Room" test, milliEgo maintained a 1.1% ATE, while visual methods failed completely.
  • Generalization: When tested in a completely unseen household environment with a different user, the model still performed accurately, proving it didn't just "memorize" the training buildings.

Limitations and Future Work

  • Outdoor Generalization: The study focused primarily on indoor commercial buildings. Performance in wide-open outdoor spaces or varied weather (rain/snow) remains to be fully explored.
  • Drift over Time: Like all odometry systems, milliEgo still drifts eventually. The authors suggest adding Loop Closure (recognizing a place you've been before) to create a full SLAM (Simultaneous Localization and Mapping) system.
  • Dynamic Platforms: Future work is needed to see if the system can handle the rapid, high-parallax motion of UAVs (drones).
  • Energy Optimization: While fast, there is still room to optimize the neural network for even lower power consumption on mobile devices.

Summary updated

Abstract

Robust and accurate trajectory estimation of mobile agents such as people and robots is a key requirement for providing spatial awareness for emerging capabilities such as augmented reality or autonomous interaction. Although currently dominated by optical techniques e.g., visual-inertial odometry, these suffer from challenges with scene illumination or featureless surfaces. As an alternative, we propose milliEgo, a novel deep-learning approach to robust egomotion estimation which exploits the capabilities of low-cost mmWave radar. Although mmWave radar has a fundamental advantage over monocular cameras of being metric i.e., providing absolute scale or depth, current single chip solutions have limited and sparse imaging resolution, making existing point-cloud registration techniques brittle. We propose a new architecture that is optimized for solving this challenging pose transformation problem. Secondly, to robustly fuse mmWave pose estimates with additional sensors, e.g. inertial or visual sensors we introduce a mixed attention approach to deep fusion. Through extensive experiments, we demonstrate our proposed system is able to achieve 1.3% 3D error drift and generalizes well to unseen environments. We also show that the neural architecture can be made highly efficient and suitable for real-time embedded applications.
BibTeX
@inproceedings{lu2020milliego,
  author = {Chris Xiaoxuan Lu and Muhamad Risqi Utama Saputra and Peijun Zhao and Yasin Almalioglu and Pedro P. B. de Gusmao and Changhao Chen and Ke Sun and Niki Trigoni and Andrew Markham},
  title = {{milliEgo: single-chip mmWave radar aided egomotion estimation via deep sensor fusion}},
  booktitle = {SenSys},
  year = {2020},
  doi = {10.1145/3384419.3430776},
}

Loading related content...

Figure