Learning Robust Features for Planar Object Tracking

Lin Chen, Yaowu Chen, Haibin Ling, Xiang Tian, Yuesong Tian

Planar object tracking—the task of estimating the 8-degree-of-freedom motion of a flat surface across video frames—is a cornerstone of augmented reality and robotic navigation. Historically, the field has been divided between feature-based methods, which are robust to large displacements but sensitive to blur, and direct methods, which offer sub-pixel precision but fail under changing illumination or noise. This paper introduces a novel end-to-end framework that bridges this gap by embedding a differentiable optimization layer within a Siamese neural network.

The ESM-Siamese Framework

The core innovation of this research is the ESM-Siamese network. Unlike standard deep learning trackers that simply regress a bounding box, this architecture is designed to learn deep features specifically optimized for a second-order mathematical solver.

The ESM-Siamese network architecture, showing the dual ResNet-18 branches and the differentiable ESM layer for homography estimation.

Key Components:

  • Siamese Backbone: The network utilizes two identical branches with shared weights to extract feature maps from a "template" (the target object) and the current video frame. The authors employ a shallow version of ResNet-18. While deeper networks are better for high-level object recognition, shallow layers preserve the high-frequency spatial details—like edges and textures—essential for precise geometric alignment.
  • Differentiable ESM Layer: The authors implemented the classical Efficient Second-order Minimization (ESM) algorithm as a differentiable layer. This allows the network to use backpropagation to learn feature representations that specifically minimize the error within the ESM optimization process.
  • Spatial Transformation Layer: This component handles the warping of feature maps based on estimated homography, allowing for iterative refinement of the object's position, scale, and perspective.

Training with Geometric Awareness

To train the model, the researchers used a massive synthetic dataset derived from MS-COCO, applying random warps, extreme lighting shifts, and motion blur to simulate real-world conditions. A critical insight in their training process was the use of Corner Distance Loss. Rather than minimizing raw homography matrix parameters, which can vary wildly in scale, the network minimizes the Euclidean distance between the four corners of the tracked object. This provides a geometrically meaningful signal that correlates directly with tracking accuracy.

Performance and Robustness

The proposed method was evaluated against two rigorous benchmarks: the Planar Object Tracking (POT) dataset and the Planar Object tracking with Illumination Changes (POIC) dataset.

  • Robustness to Blur and Occlusion: On the POT dataset, the ESM-Siamese method outperformed the leading direct trackers by 10% to 16% in categories involving heavy motion blur and partial occlusion.
  • Illumination Resilience: On the POIC dataset, the model achieved the highest success rate among all tested methods. By learning to ignore specular reflections and shadows—challenges that typically cause raw-pixel direct methods to fail—the deep features proved significantly more reliable.
  • Precision: Despite the added robustness, the system maintained the sub-pixel accuracy that is the hallmark of direct optimization, making it suitable for high-precision AR applications.

Conclusion and Future Directions

By making the optimization process itself differentiable, the authors have created a tracker that combines the "toughness" of deep learning with the mathematical precision of classical methods. While the system currently struggles with re-localization after a total loss of the target, future work may involve hybridizing this approach with traditional feature-matching to create a truly "unshakeable" tracker. Currently, the multi-resolution feature pyramid uses a fixed depth; making this depth adaptive to the complexity of the scene remains an open area for optimization.

Summary updated

Abstract

Visual tracking of planar objects with multiple degrees of freedom is a core component for numerous vision-based applications. Generally, since direct methods use all raw pixels in the region of interest to estimate the motion model directly, these methods are sensitive to illumination changes, partial occlusion, and motion blur. Recently, the deep convolutional network has demonstrated remarkable ability in visual tracking via learning robust deep features. In this paper, deep features are used for improving the robustness of direct methods. To learn suitable features in an end-to-end fashion, we employ a novel network architecture, the efficient second-order minimization (ESM) layer, which performs the ESM algorithm on deep feature maps. We train and validate the convolutional features on a synthetic dataset generated from the MS-COCO dataset and evaluate the tracking performance on two challenging, real-world datasets. The experimental results show that the proposed approach outperforms most state-of-the-art methods in various tracking challenges.
BibTeX
@article{chen2019learning,
  author = {Lin Chen and Yaowu Chen and Haibin Ling and Xiang Tian and Yuesong Tian},
  title = {{Learning Robust Features for Planar Object Tracking}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2924732},
}
Figure