Learning Local Feature Descriptor with Motion Attribute For Vision-based Localization

Yafei Song, Di Zhu, Jia Li, Yonghong Tian, Mingyang Li

Abstract

Vision-based localization in dynamic urban environments is often compromised by "noise" from moving objects like vehicles and pedestrians. This paper introduces MD-Net, a multi-task neural network that extracts local feature descriptors while simultaneously predicting motion attributes (static, moving, or unstable). By proactively filtering out features associated with non-static regions, MD-Net enhances the robustness and accuracy of localization systems. Experimental results on benchmarks like Cityscapes and HPatches, as well as real-world urban tests, demonstrate that MD-Net significantly reduces positional drift compared to traditional methods.

Introduction

Reliable vision-based localization—the process by which a robot or vehicle determines its position using camera data—is a cornerstone of autonomous navigation. Most existing algorithms rely on matching "local features" (distinct points in an image) across frames, operating under the assumption of a static world. However, real-world urban environments are inherently dynamic. Pedestrians, moving vehicles, and even parked cars that eventually move introduce significant "noise."

Anchoring a localization system to a moving object leads to catastrophic position estimate drift. While geometric constraints can filter some fast-moving objects, they often fail with "long-term moving" objects or unstable regions like the sky. MD-Net addresses this by integrating semantic motion awareness directly into the feature description process.

The MD-Net Architecture

MD-Net is a Fully Convolutional Network (FCN) designed for both efficiency and robustness. Its architecture is divided into three primary components:

  1. Shared Backbone ($N_B$): A series of convolutional layers that extract general feature maps from the input image, shared by both downstream tasks to minimize computational overhead.
  2. Motion Attribute Branch ($N_M$): This branch classifies every pixel into one of three categories: Static (e.g., buildings, roads), Moving (e.g., vehicles, humans), or Unstable (e.g., sky, vegetation). A re-weighted cross-entropy loss is employed to address the class imbalance between static and moving pixels.
  3. Descriptor Branch ($N_D$): This branch generates a 128-dimensional descriptor vector for each point. To achieve state-of-the-art performance without exhaustive new datasets, the authors utilized a teacher-student framework, training MD-Net to mimic the outputs of the high-performing HardNet model.
The MD-Net architecture featuring a shared backbone and dual branches for motion attribute prediction and feature description.

Feature Filtering and Localization

The core innovation of MD-Net lies in its ability to prune unreliable features before they enter the localization pipeline. During inference, a standard detector (such as FAST) identifies candidate points. MD-Net then provides both a descriptor and a motion label for each point. Points labeled as "moving" or "unstable" are discarded, ensuring that the localization algorithm only utilizes stable, static landmarks.

The operational flow of MD-Net, illustrating how local features are filtered based on predicted motion attributes.

Key Results and Insights

The effectiveness of MD-Net was validated through several benchmarks and real-world tests:

  • Motion Prediction: MD-Net achieved a mean Intersection over Union (mIoU) of 76.2% on the Cityscapes validation set, demonstrating high accuracy in identifying potentially dynamic regions.
  • Descriptor Quality: On the HPatches benchmark, MD-Net outperformed other FCN-based methods like SuperPoint and matched the performance of HardNet, proving that multi-tasking does not degrade descriptor quality.
  • Localization Accuracy: In real-world urban tests involving a car equipped with high-precision RTK-GPS, MD-Net significantly outperformed standard methods. In environments dense with moving traffic, MD-Net achieved a root-mean-squared error (RMSE) of 1.10m, whereas traditional methods like ORB or FREAK often failed or exhibited errors exceeding 10m.

Conclusion

MD-Net demonstrates that semantic motion attributes can serve as a powerful proxy for identifying reliable features in dynamic environments. By sharing a backbone, the system provides motion-aware filtering with minimal additional computational cost, making it suitable for real-time robotics. Future work aims to integrate the feature detector directly into the FCN for a fully end-to-end system and to expand training data to improve generalization across diverse environments.

Summary updated

BibTeX
@inproceedings{song2019learning,
  author = {Yafei Song and Di Zhu and Jia Li and Yonghong Tian and Mingyang Li},
  title = {{Learning Local Feature Descriptor with Motion Attribute For Vision-based Localization}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967749},
}
Figure