milliEgo: mmWave Aided Egomotion Estimation with Deep Sensor Fusion

Chris Xiaoxuan Lu, Muhamad Risqi Utama Saputra, Peijun Zhao, Yasin Almalioglu, Pedro P. B. de Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, Andrew Markham

Abstract: Robust and accurate trajectory estimation of mobile agents such as people and robots is a key requirement for providing spatial awareness for emerging capabilities such as augmented reality or autonomous interaction. Although currently dominated by optical techniques e.g., visual-inertial odometry, these suffer from challenges with scene illumination or featureless surfaces. As an alternative, we propose milliEgo, a novel deep-learning approach to robust egomotion estimation which exploits the capabilities of low-cost mmWave radar. Although mmWave radar has a fundamental advantage over monocular cameras of being metric i.e., providing absolute scale or depth, current single chip solutions have limited and sparse imaging resolution, making existing point-cloud registration techniques brittle. We propose a new architecture that is optimized for solving this challenging pose transformation problem. Secondly, to robustly fuse mmWave pose estimates with additional sensors, e.g. inertial or visual sensors we introduce a mixed attention approach to deep fusion. Through extensive experiments, we demonstrate our proposed system is able to achieve 1.3% 3D error drift and generalizes well to unseen environments. We also show that the neural architecture can be made highly efficient and suitable for real-time embedded applications.

The Challenge of Sparse Sensing

Accurate egomotion estimation—knowing exactly how a robot or person is moving—is the backbone of autonomous navigation and AR/VR. Currently, most systems rely on Visual-Inertial Odometry (VIO). While effective, VIO fails in the dark, in blinding light, or in "featureless" areas like plain white hallways. Lidar is a robust alternative but remains too bulky and expensive for small robots or smartphones.

Millimeter-wave (mmWave) radar is cheap, small, and works in any lighting or weather. However, the data it produces is significantly more challenging than Lidar. A typical mmWave scan has 100x fewer points than a Lidar scan, and up to 75% of the detected points can be "ghost" reflections caused by multi-path interference. Traditional geometric algorithms, such as Iterative Closest Point (ICP), typically break down when faced with such low-quality, sparse data.

Comparison of Lidar (a) and mmWave (b) point clouds at consecutive time steps (t-1, t), illustrating the extreme sparsity and noise inherent in mmWave data.

The milliEgo Architecture

milliEgo treats egomotion as a learning problem rather than a purely geometric one. The system utilizes a deep-learning pipeline designed to extract meaningful motion features from messy sensor data.

To make unstructured 3D radar points digestible for a neural network, the system projects them onto a 2D "depth" image using a panoramic view. This allows the use of powerful Convolutional Neural Networks (CNNs) to extract motion features. The core of the system is a two-stage "Mixed Attention" fusion mechanism:

  1. Stage 1 (Intra-sensor): Each sensor filters its own noise (e.g., the radar ignoring multi-path reflections).
  2. Stage 2 (Inter-sensor): The sensors "cross-talk" to decide which one is more reliable at any given moment. If the IMU is glitching or the radar has low visibility, the model adaptively re-weights their contributions.

Finally, a two-layer LSTM (Long Short-Term Memory) network tracks the movement over time, ensuring the trajectory remains smooth and consistent.

The milliEgo architecture, showing the mmWave and IMU subnets, the multi-modal fusion block, and the temporal modeling stage leading to 6-DoF pose regression.

Experimental Evaluation and Results

The researchers collected a massive dataset covering over 8km of trajectories across 17 distinct floors in 6 different commercial buildings. The environments included hallways, canteens, and cluttered storerooms, specifically targeting areas with glass walls and dynamic lighting that often confuse optical sensors.

Key results include:

  • High Accuracy: milliEgo achieved a remarkably low 1.3% 3D distance drift. In standard tests, it maintained a mean Absolute Trajectory Error (ATE) of just 0.814 meters.
  • Environmental Robustness: In a "Dark Room" test, milliEgo maintained a 1.1% ATE, while visual methods failed completely. It also significantly outperformed baseline methods like VINET (Visual-Inertial) in glass-heavy environments.
  • Real-time Efficiency: The model is optimized for embedded hardware, running at 17 FPS on an NVIDIA Jetson TX2 with a footprint of only 107MB.

Conclusion and Future Work

milliEgo demonstrates that deep learning can successfully bridge the gap between low-cost, sparse mmWave sensors and high-precision egomotion requirements. While the current system focuses on indoor environments, future work will explore outdoor generalization and the addition of loop closure to evolve the system into a full Simultaneous Localization and Mapping (SLAM) solution. The efficiency of the architecture makes it a promising candidate for real-time spatial awareness in smartphones, AR/VR headsets, and small-scale robotics.

Summary updated

Abstract

Robust and accurate trajectory estimation of mobile agents such as people and robots is a key requirement for providing spatial awareness for emerging capabilities such as augmented reality or autonomous interaction. Although currently dominated by optical techniques e.g., visual-inertial odometry, these suffer from challenges with scene illumination or featureless surfaces. As an alternative, we propose milliEgo, a novel deep-learning approach to robust egomotion estimation which exploits the capabilities of low-cost mmWave radar. Although mmWave radar has a fundamental advantage over monocular cameras of being metric i.e., providing absolute scale or depth, current single chip solutions have limited and sparse imaging resolution, making existing point-cloud registration techniques brittle. We propose a new architecture that is optimized for solving this challenging pose transformation problem. Secondly, to robustly fuse mmWave pose estimates with additional sensors, e.g. inertial or visual sensors we introduce a mixed attention approach to deep fusion. Through extensive experiments, we demonstrate our proposed system is able to achieve 1.3% 3D error drift and generalizes well to unseen environments. We also show that the neural architecture can be made highly efficient and suitable for real-time embedded applications.
BibTeX
@inproceedings{lu2020milliego,
  author = {Chris Xiaoxuan Lu and Muhamad Risqi Utama Saputra and Peijun Zhao and Yasin Almalioglu and Pedro P. B. de Gusmao and Changhao Chen and Ke Sun and Niki Trigoni and Andrew Markham},
  title = {{milliEgo: mmWave Aided Egomotion Estimation with Deep Sensor Fusion}},
  booktitle = {SenSys},
  year = {2020},
  doi = {10.1145/3384419.3430776},
}
Figure