Abstract
Vision-based localization in dynamic urban environments is often compromised by "noise" from moving objects like vehicles and pedestrians. This paper introduces MD-Net, a multi-task neural network that extracts local feature descriptors while simultaneously predicting motion attributes (static, moving, or unstable). By proactively filtering out features associated with non-static regions, MD-Net enhances the robustness and accuracy of localization systems. Experimental results on benchmarks like Cityscapes and HPatches, as well as real-world urban tests, demonstrate that MD-Net significantly reduces positional drift compared to traditional methods.
Introduction
Reliable vision-based localization—the process by which a robot or vehicle determines its position using camera data—is a cornerstone of autonomous navigation. Most existing algorithms rely on matching "local features" (distinct points in an image) across frames, operating under the assumption of a static world. However, real-world urban environments are inherently dynamic. Pedestrians, moving vehicles, and even parked cars that eventually move introduce significant "noise."
Anchoring a localization system to a moving object leads to catastrophic position estimate drift. While geometric constraints can filter some fast-moving objects, they often fail with "long-term moving" objects or unstable regions like the sky. MD-Net addresses this by integrating semantic motion awareness directly into the feature description process.
The MD-Net Architecture
MD-Net is a Fully Convolutional Network (FCN) designed for both efficiency and robustness. Its architecture is divided into three primary components:
- Shared Backbone ($N_B$): A series of convolutional layers that extract general feature maps from the input image, shared by both downstream tasks to minimize computational overhead.
- Motion Attribute Branch ($N_M$): This branch classifies every pixel into one of three categories: Static (e.g., buildings, roads), Moving (e.g., vehicles, humans), or Unstable (e.g., sky, vegetation). A re-weighted cross-entropy loss is employed to address the class imbalance between static and moving pixels.
- Descriptor Branch ($N_D$): This branch generates a 128-dimensional descriptor vector for each point. To achieve state-of-the-art performance without exhaustive new datasets, the authors utilized a teacher-student framework, training MD-Net to mimic the outputs of the high-performing HardNet model.
Feature Filtering and Localization
The core innovation of MD-Net lies in its ability to prune unreliable features before they enter the localization pipeline. During inference, a standard detector (such as FAST) identifies candidate points. MD-Net then provides both a descriptor and a motion label for each point. Points labeled as "moving" or "unstable" are discarded, ensuring that the localization algorithm only utilizes stable, static landmarks.
Key Results and Insights
The effectiveness of MD-Net was validated through several benchmarks and real-world tests:
- Motion Prediction: MD-Net achieved a mean Intersection over Union (mIoU) of 76.2% on the Cityscapes validation set, demonstrating high accuracy in identifying potentially dynamic regions.
- Descriptor Quality: On the HPatches benchmark, MD-Net outperformed other FCN-based methods like SuperPoint and matched the performance of HardNet, proving that multi-tasking does not degrade descriptor quality.
- Localization Accuracy: In real-world urban tests involving a car equipped with high-precision RTK-GPS, MD-Net significantly outperformed standard methods. In environments dense with moving traffic, MD-Net achieved a root-mean-squared error (RMSE) of 1.10m, whereas traditional methods like ORB or FREAK often failed or exhibited errors exceeding 10m.
Conclusion
MD-Net demonstrates that semantic motion attributes can serve as a powerful proxy for identifying reliable features in dynamic environments. By sharing a backbone, the system provides motion-aware filtering with minimal additional computational cost, making it suitable for real-time robotics. Future work aims to integrate the feature detector directly into the FCN for a fully end-to-end system and to expand training data to improve generalization across diverse environments.
Summary updated