Pedestrian tracking framework based on deep convolution network and SIFT

Jing Liu, Kai Chen, Xiao SongScientia 2018 · DOI

Abstract

Numerous object-tracking and multiple people-tracking algorithms have been put forward in the field of computer vision, but these algorithms did not address the issue of when a pedestrian is partially or fully occluded by another object or person. To achieve efficient pedestrian tracking in various occlusion conditions, this research presents a pedestrian-tracking framework based on deep learning. First, a pedestrian detector was trained as a pedestrian-tracking search mechanism based on the object detection algorithm Faster R-CNN, which narrows the search range and improves accuracy efficiently compared with traditional gradient-down algorithms. Second, color histogram and scale-invariant features transform (SIFT) were combined as the target model expression. In the process of target matching, a full convolution network (FCN) was trained for pedestrians to extract the pedestrian information in the target model base on an FCN image semantic segmentation algorithm for removing the background noise. Finally, extensive experiments on OTB demonstrate that the proposed method achieves better performance than other state-of-the-art trackers for precision and success rate.

Introduction

In the fields of autonomous driving and intelligent surveillance, robust pedestrian tracking is essential. However, maintaining a target's identity through complex scenes remains a significant challenge, particularly when pedestrians are partially or fully occluded by obstacles or other people. Traditional tracking algorithms, such as Mean Shift or Kernelized Correlation Filters (KCF), typically rely on local search mechanisms that track targets based on their previous positions. These methods often fail during occlusion events, leading to "drift" where the tracker locks onto the occluding object or loses the target entirely.

Proposed Methodology

This research introduces a hybrid tracking framework that integrates deep learning architectures with classical computer vision descriptors to overcome the limitations of local-search trackers.

1. Detection-based Search with Faster R-CNN

Instead of relying on simple gradient descent or local neighborhood searches, the framework employs a pre-trained Faster R-CNN as a search mechanism. This allows the system to detect all potential pedestrian candidates within a frame, providing a robust "pool" for matching even if the target has moved significantly or reappeared after total occlusion.

The Faster R-CNN architecture serves as the core detection mechanism for identifying pedestrian candidates.

2. Semantic Masking via Fully Convolutional Networks (FCN)

A key innovation in this framework is the use of a Fully Convolutional Network (FCN), based on the VGG16 architecture, to perform semantic segmentation. By segmenting the pedestrian from the background within the detected bounding box, the system can effectively strip away background noise (e.g., trees, walls). This ensures that subsequent feature extraction is focused solely on the target pedestrian.

3. Dual-Feature Representation and Hierarchical Matching

The target model is expressed through a combination of color histograms and Scale-Invariant Feature Transform (SIFT) descriptors:

  • Color Histograms: Provide a computationally efficient first pass for similarity checking.
  • SIFT: Used when color matching is insufficient, such as during partial occlusion. SIFT is highly robust to changes in scale, rotation, and illumination.

The algorithm follows a hierarchical logic: it first attempts matching using color similarity. If the similarity score falls below a dynamic threshold—indicating a complex scene or occlusion—the system triggers the more intensive SIFT and FCN pipeline to identify the best candidate.

Experimental Evaluation

The Faster R-CNN component was trained on the VOC2012 dataset, augmented with specific data to improve robustness against partial occlusion. The complete framework was evaluated using the Object Tracking Benchmark (OTB), focusing on sequences characterized by heavy occlusion, such as "Jogging" and "Multi-pedestrian Occlusion."

Key Results

  • Performance: The proposed method consistently achieved higher precision and success rates compared to state-of-the-art trackers like KCF, STC, and CSK.
  • Occlusion Recovery: In the "Pedestrian Occlusion" sequence, the framework successfully re-acquired the target at frame 73 after it was lost at frame 63, whereas traditional trackers failed to recover.
  • Differentiation: The use of FCN-based masking significantly improved the "differentiation degree," allowing the tracker to better distinguish the target from other nearby pedestrians.

Conclusion and Future Work

While the framework demonstrates superior robustness in occlusion scenarios, its performance is currently sensitive to image resolution; low-resolution targets provide insufficient data for effective SIFT and FCN processing. Future research will focus on optimizing the computational efficiency of the deep learning components to improve real-time performance and exploring advanced target representations to handle extreme deformations and lighting variations.

Summary updated

Abstract

Numerous object-tracking and multiple people-tracking algorithms have been put forward in the field of computer vision, but these algorithms did not address the issue of when a pedestrian is partially or fully occluded by another object or person. To achieve efficient pedestrian tracking in various occlusion conditions, this research presents a pedestrian-tracking framework based on deep learning. First, a pedestrian detector was trained as a pedestrian-tracking search mechanism based on the object detection algorithm Faster R-CNN, which narrows the search range and improves accuracy efficiently compared with traditional gradient-down algorithms. Second, color histogram and scale-invariant features transform (SIFT) were combined as the target model expression. In the process of target matching, a full convolution network (FCN) was trained for pedestrians to extract the pedestrian information in the target model base on an FCN image semantic segmentation algorithm for removing the background noise. Finally, extensive experiments on OTB demonstrate that the proposed method achieves better performance than other state-of-the-art trackers for precision and success rate.
BibTeX
@inproceedings{liu2018pedestrian,
  author = {Jing Liu and Kai Chen and Xiao SONG},
  title = {{Pedestrian tracking framework based on deep convolution network and SIFT}},
  booktitle = {Scientia},
  year = {2018},
  doi = {10.1360/n112017-00286},
}
Figure