Abstract: Robust and accurate trajectory estimation of mobile agents such as people and robots is a key requirement for providing spatial awareness for emerging capabilities such as augmented reality or autonomous interaction. Although currently dominated by optical techniques e.g., visual-inertial odometry, these suffer from challenges with scene illumination or featureless surfaces. As an alternative, we propose milliEgo, a novel deep-learning approach to robust egomotion estimation which exploits the capabilities of low-cost mmWave radar. Although mmWave radar has a fundamental advantage over monocular cameras of being metric i.e., providing absolute scale or depth, current single chip solutions have limited and sparse imaging resolution, making existing point-cloud registration techniques brittle. We propose a new architecture that is optimized for solving this challenging pose transformation problem. Secondly, to robustly fuse mmWave pose estimates with additional sensors, e.g. inertial or visual sensors we introduce a mixed attention approach to deep fusion. Through extensive experiments, we demonstrate our proposed system is able to achieve 1.3% 3D error drift and generalizes well to unseen environments. We also show that the neural architecture can be made highly efficient and suitable for real-time embedded applications.
The Challenge of Sparse Sensing
Accurate egomotion estimation—knowing exactly how a robot or person is moving—is the backbone of autonomous navigation and AR/VR. Currently, most systems rely on Visual-Inertial Odometry (VIO). While effective, VIO fails in the dark, in blinding light, or in "featureless" areas like plain white hallways. Lidar is a robust alternative but remains too bulky and expensive for small robots or smartphones.
Millimeter-wave (mmWave) radar is cheap, small, and works in any lighting or weather. However, the data it produces is significantly more challenging than Lidar. A typical mmWave scan has 100x fewer points than a Lidar scan, and up to 75% of the detected points can be "ghost" reflections caused by multi-path interference. Traditional geometric algorithms, such as Iterative Closest Point (ICP), typically break down when faced with such low-quality, sparse data.
The milliEgo Architecture
milliEgo treats egomotion as a learning problem rather than a purely geometric one. The system utilizes a deep-learning pipeline designed to extract meaningful motion features from messy sensor data.
To make unstructured 3D radar points digestible for a neural network, the system projects them onto a 2D "depth" image using a panoramic view. This allows the use of powerful Convolutional Neural Networks (CNNs) to extract motion features. The core of the system is a two-stage "Mixed Attention" fusion mechanism:
- Stage 1 (Intra-sensor): Each sensor filters its own noise (e.g., the radar ignoring multi-path reflections).
- Stage 2 (Inter-sensor): The sensors "cross-talk" to decide which one is more reliable at any given moment. If the IMU is glitching or the radar has low visibility, the model adaptively re-weights their contributions.
Finally, a two-layer LSTM (Long Short-Term Memory) network tracks the movement over time, ensuring the trajectory remains smooth and consistent.
Experimental Evaluation and Results
The researchers collected a massive dataset covering over 8km of trajectories across 17 distinct floors in 6 different commercial buildings. The environments included hallways, canteens, and cluttered storerooms, specifically targeting areas with glass walls and dynamic lighting that often confuse optical sensors.
Key results include:
- High Accuracy: milliEgo achieved a remarkably low 1.3% 3D distance drift. In standard tests, it maintained a mean Absolute Trajectory Error (ATE) of just 0.814 meters.
- Environmental Robustness: In a "Dark Room" test, milliEgo maintained a 1.1% ATE, while visual methods failed completely. It also significantly outperformed baseline methods like VINET (Visual-Inertial) in glass-heavy environments.
- Real-time Efficiency: The model is optimized for embedded hardware, running at 17 FPS on an NVIDIA Jetson TX2 with a footprint of only 107MB.
Conclusion and Future Work
milliEgo demonstrates that deep learning can successfully bridge the gap between low-cost, sparse mmWave sensors and high-precision egomotion requirements. While the current system focuses on indoor environments, future work will explore outdoor generalization and the addition of loop closure to evolve the system into a full Simultaneous Localization and Mapping (SLAM) solution. The efficiency of the architecture makes it a promising candidate for real-time spatial awareness in smartphones, AR/VR headsets, and small-scale robotics.
Summary updated