TL;DR
milliEgo is the first deep-learning-based odometry system that uses low-cost, single-chip mmWave radar and IMU data to track a device's movement (egomotion) in 3D space. By transforming sparse, noisy radar point clouds into structured images and using a novel "mixed attention" fusion mechanism, it achieves a remarkably low 1.3% distance drift, outperforming traditional visual and inertial methods in challenging environments like dark rooms or glass corridors.
What Problem Does This Paper Address?
Accurate egomotion estimation—knowing exactly how a robot or person is moving—is the backbone of autonomous navigation and AR/VR. Currently, most systems rely on Visual-Inertial Odometry (VIO). While effective, VIO fails in the dark, in blinding light, or in "featureless" areas (like a hallway with plain white walls). Lidar is a robust alternative but remains too bulky and expensive for small robots or smartphones.
Millimeter-wave (mmWave) radar is cheap, small, and works in any lighting or weather. However, the data it produces is "messy":
- Extreme Sparsity: A typical mmWave scan has 100x fewer points than a Lidar scan.
- High Noise: Up to 75% of the detected points can be "ghost" reflections caused by multi-path interference (signals bouncing off multiple surfaces). Traditional geometric algorithms (like Iterative Closest Point) simply break down when faced with such low-quality data.
Methods
The authors propose a deep-learning pipeline that treats egomotion as a learning problem rather than a purely geometric one.
- Panoramic Projection: To make unstructured 3D radar points digestible for a neural network, they project the points onto a 2D "depth" image using a panoramic view. This allows them to use powerful Convolutional Neural Networks (CNNs) to extract motion features.
- Mixed Attention Fusion (The "Secret Sauce"): Instead of just concatenating radar and IMU data, they use a two-stage attention mechanism:
- Stage 1 (Intra-sensor): Each sensor "looks at itself" to filter out its own noise (e.g., the radar ignoring multi-path reflections).
- Stage 2 (Inter-sensor): The sensors "cross-talk" to decide which one is more reliable at any given moment. If the IMU is glitching or the radar has low visibility, the model adaptively re-weights their contributions.
- Temporal Modeling: A two-layer LSTM (Long Short-Term Memory) network tracks the movement over time, ensuring the trajectory remains smooth and consistent.
Datasets
The researchers collected a massive, diverse dataset to prove their system's robustness:
- Scale: Over 8km of trajectories across 17 distinct floors in 6 different commercial buildings.
- Hardware: Data was collected using both a mobile robot (Turtlebot 2) and a custom 3D-printed handheld device.
- Ground Truth: They used high-end, "bulky" Lidar sensors to provide the "perfect" trajectory for training and evaluation.
- Diversity: The environments included hallways, canteens, atrium areas, and cluttered storerooms, specifically targeting areas with glass walls and dynamic lighting that confuse cameras.
Key Insights and Ideas
- Learning beats Geometry for "Bad" Data: Traditional math-heavy registration fails on sparse radar data because there aren't enough points to "match." Deep learning can learn the underlying patterns of motion even when the data is incomplete.
- The Power of "Cross-talk": The authors found that single-stage attention (common in other models) makes sensors change weights synchronously. Their two-stage "Mixed Attention" allows sensors to dominate interchangeably—for example, the IMU takes over during sharp turns where radar might struggle with parallax.
- Efficiency is Possible: Despite using complex DNNs, the model is small enough (107MB) to run at 17 FPS on an NVIDIA Jetson TX2, making it viable for real-time use on embedded hardware.
Key Results
- Accuracy: milliEgo achieved a 1.3% 3D error drift. In a typical test, it had a mean Absolute Trajectory Error (ATE) of just 0.814 meters.
- Superiority: It significantly outperformed baseline methods, including VINET (Visual-Inertial) and DINET (Depth-Inertial), especially in environments with glass or poor lighting.
- Robustness: In a "Dark Room" test, milliEgo maintained a 1.1% ATE, while visual methods failed completely.
- Generalization: When tested in a completely unseen household environment with a different user, the model still performed accurately, proving it didn't just "memorize" the training buildings.
Limitations and Future Work
- Outdoor Generalization: The study focused primarily on indoor commercial buildings. Performance in wide-open outdoor spaces or varied weather (rain/snow) remains to be fully explored.
- Drift over Time: Like all odometry systems, milliEgo still drifts eventually. The authors suggest adding Loop Closure (recognizing a place you've been before) to create a full SLAM (Simultaneous Localization and Mapping) system.
- Dynamic Platforms: Future work is needed to see if the system can handle the rapid, high-parallax motion of UAVs (drones).
- Energy Optimization: While fast, there is still room to optimize the neural network for even lower power consumption on mobile devices.
Summary updated