Learning Local Feature Descriptor with Motion Attribute for Vision-based Localization

Yafei Song, Di Zhu, Jia Li, Yonghong Tian, Mingyang Li

Abstract

Vision-based localization in dynamic urban environments is often compromised by "noise" from moving objects like vehicles and pedestrians. This paper introduces MD-Net, a multi-task neural network that extracts local feature descriptors while simultaneously predicting motion attributes (static, moving, or unstable). By proactively filtering out features associated with non-static regions, MD-Net enhances the robustness and accuracy of localization systems. Experimental results on benchmarks like Cityscapes and HPatches, as well as real-world urban tests, demonstrate that MD-Net significantly reduces positional drift compared to traditional methods.

Introduction

Reliable vision-based localization—the process by which a robot or vehicle determines its position using camera data—is a cornerstone of autonomous navigation. Most existing algorithms rely on matching "local features" (distinct points in an image) across frames, operating under the assumption of a static world. However, real-world urban environments are inherently dynamic. Pedestrians, moving vehicles, and even parked cars that eventually move introduce significant "noise."

Anchoring a localization system to a moving object leads to catastrophic position estimate drift. While geometric constraints can filter some fast-moving objects, they often fail with "long-term moving" objects or unstable regions like the sky. MD-Net addresses this by integrating semantic motion awareness directly into the feature description process.

The MD-Net Architecture

MD-Net is a Fully Convolutional Network (FCN) designed for both efficiency and robustness. Its architecture is divided into three primary components:

  1. Shared Backbone ($N_B$): A series of convolutional layers that extract general feature maps from the input image, shared by both downstream tasks to minimize computational overhead.
  2. Motion Attribute Branch ($N_M$): This branch classifies every pixel into one of three categories: Static (e.g., buildings, roads), Moving (e.g., vehicles, humans), or Unstable (e.g., sky, vegetation). A re-weighted cross-entropy loss is employed to address the class imbalance between static and moving pixels.
  3. Descriptor Branch ($N_D$): This branch generates a 128-dimensional descriptor vector for each point. To achieve state-of-the-art performance without exhaustive new datasets, the authors utilized a teacher-student framework, training MD-Net to mimic the outputs of the high-performing HardNet model.
The MD-Net architecture featuring a shared backbone and dual branches for motion attribute prediction and feature description.

Feature Filtering and Localization

The core innovation of MD-Net lies in its ability to prune unreliable features before they enter the localization pipeline. During inference, a standard detector (such as FAST) identifies candidate points. MD-Net then provides both a descriptor and a motion label for each point. Points labeled as "moving" or "unstable" are discarded, ensuring that the localization algorithm only utilizes stable, static landmarks.

The operational flow of MD-Net, illustrating how local features are filtered based on predicted motion attributes.

Key Results and Insights

The effectiveness of MD-Net was validated through several benchmarks and real-world tests:

  • Motion Prediction: MD-Net achieved a mean Intersection over Union (mIoU) of 76.2% on the Cityscapes validation set, demonstrating high accuracy in identifying potentially dynamic regions.
  • Descriptor Quality: On the HPatches benchmark, MD-Net outperformed other FCN-based methods like SuperPoint and matched the performance of HardNet, proving that multi-tasking does not degrade descriptor quality.
  • Localization Accuracy: In real-world urban tests involving a car equipped with high-precision RTK-GPS, MD-Net significantly outperformed standard methods. In environments dense with moving traffic, MD-Net achieved a root-mean-squared error (RMSE) of 1.10m, whereas traditional methods like ORB or FREAK often failed or exhibited errors exceeding 10m.

Conclusion

MD-Net demonstrates that semantic motion attributes can serve as a powerful proxy for identifying reliable features in dynamic environments. By sharing a backbone, the system provides motion-aware filtering with minimal additional computational cost, making it suitable for real-time robotics. Future work aims to integrate the feature detector directly into the FCN for a fully end-to-end system and to expand training data to improve generalization across diverse environments.

Summary updated

Abstract

In recent years, camera-based localization has been widely used for robotic applications, and most proposed algorithms rely on local features extracted from recorded images. For better performance, the features used for open-loop localization are required to be short-term globally static, and the ones used for re-localization or loop closure detection need to be long-term static. Therefore, the motion attribute of a local feature point could be exploited to improve localization performance, e.g., the feature points extracted from moving persons or vehicles can be excluded from these systems due to their unsteadiness. In this paper, we design a fully convolutional network (FCN), named MD-Net, to perform motion attribute estimation and feature description simultaneously. MD-Net has a shared backbone network to extract features from the input image and two network branches to complete each sub-task. With MD-Net, we can obtain the motion attribute while avoiding increasing much more computation. Experimental results demonstrate that the proposed method can learn distinct local feature descriptor along with motion attribute only using an FCN, by outperforming competing methods by a wide margin. We also show that the proposed algorithm can be integrated into a vision-based localization algorithm to improve estimation accuracy significantly.
BibTeX
@inproceedings{song2019learning,
  author = {Yafei Song and Di Zhu and Jia Li and Yonghong Tian and Mingyang Li},
  title = {{Learning Local Feature Descriptor with Motion Attribute for Vision-based Localization}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8967749},
}
Figure