Multiple Object Tracking via Feature Pyramid Siamese Networks

Sangyun Lee, Euntai Kim

Abstract

When multiple object tracking (MOT) based on the tracking-by-detection paradigm is implemented, the similarity metric between the current detections and existing tracks plays an essential role. Most of the MOT schemes based on a deep neural network learn the similarity metric using a Siamese architecture, but the plain Siamese architecture might not be enough owing to its structural simplicity and lack of motion information. This paper aims to propose a new MOT scheme to overcome the existing problems in the conventional MOTs. Feature pyramid Siamese network (FPSN) is proposed to address the structural simplicity. The FPSN is inspired by a feature pyramid network (FPN) and it extends the Siamese network by applying FPN to the plain Siamese architecture and by developing a new multi-level discriminative feature. A spatiotemporal motion feature is added to the FPSN to overcome the lack of motion information and to enhance the performance in MOT. Thus, FPSN-MOT considers not only the appearance feature but also motion information. Finally, FPSN-MOT is applied to the public MOT challenge benchmark problems and its performance is compared to that of the other state-of-the-art MOT methods.

Introduction

In the "tracking-by-detection" paradigm, the primary challenge lies in data association: correctly linking new detections in the current frame to existing tracks from previous frames. While modern systems often utilize Siamese networks for appearance matching, standard architectures typically rely on a single high-level feature layer. This can be insufficient for distinguishing between visually similar objects, such as individuals in similar clothing. Furthermore, the omission of motion information—physical constraints on object movement—often leads to "ID switches" during occlusions or in crowded scenes.

Methodology: Feature Pyramid Siamese Network (FPSN)

The authors propose the Feature Pyramid Siamese Network (FPSN) to enhance discriminative power. Inspired by Feature Pyramid Networks (FPN) used in object detection, FPSN extracts features at multiple scales. It utilizes a backbone network (such as SqueezeNet or Inception) and incorporates a top-down pathway with deconvolution layers to fuse high-level semantic information with low-level structural details.

Architecture of the Feature Pyramid Siamese Network showing multi-level feature extraction.

The resulting multi-level feature vector is significantly more discriminative than single-layer outputs. To train this architecture, a margin-based multi-level contrastive loss is applied across all pyramid levels, forcing the network to learn similarities and differences across all scales simultaneously.

FPSN-MOT Framework

The complete tracking system, FPSN-MOT, integrates these multi-level appearance features with a 12-dimensional spatiotemporal motion vector. This vector captures relative displacement, bounding box size changes, and time gaps between detections and tracks.

Overview of the FPSN-MOT tracking framework.

As shown in the system flowchart, detections and targets are first filtered through a gating mechanism. The selected image pairs are processed by the FPSN to extract multi-level features, which are then concatenated with spatiotemporal motion features. A binary classifier produces a final similarity score used for target association and management.

Key Insights and Implementation Details

Several design choices contribute to the effectiveness of the FPSN-MOT system:

  • Deconvolution for Upsampling: Unlike the original FPN which uses simple nearest-neighbor upsampling, this paper employs deconvolution layers. Learned upsampling reduces aliasing and distortion, which is critical for fine-grained visual verification tasks.
  • Motion as a Tie-Breaker: In crowded scenes where two individuals might look nearly identical, the appearance model alone may struggle. The motion feature acts as a tie-breaker, ensuring the track follows a physically plausible trajectory.
  • Global Average Pooling (GAP): By using GAP to vectorize the pyramidal feature maps, the model avoids the overfitting common in fully connected layers and reduces memory requirements, making it more practical for real-time applications.

Evaluation and Results

The system was evaluated on several rigorous benchmarks:

  • Person Re-Identification: FPSN outperformed baseline Siamese networks (which only use the last feature layer) across Recall, Precision, F1, and Accuracy metrics.
  • MOT17 Challenge: FPSN-MOT achieved a MOTA (Multiple Object Tracking Accuracy) of 44.9 and an IDF1 score of 48.4, surpassing other referenceable online tracking methods at the time of publication.
  • KITTI Tracking: Ablation studies demonstrated that motion features are crucial for maintaining correct IDs in scenarios where appearance-only models failed, such as two similar cars driving closely together.

Conclusion

The FPSN-MOT approach demonstrates that combining multi-scale appearance features with spatiotemporal motion information significantly improves tracking robustness. While performance remains dependent on the quality of initial detections, the structural improvements of the FPSN provide a more discriminative foundation for modern MOT tasks.

Summary updated

Abstract

When multiple object tracking (MOT) based on the tracking-by-detection paradigm is implemented, the similarity metric between the current detections and existing tracks plays an essential role. Most of the MOT schemes based on a deep neural network learn the similarity metric using a Siamese architecture, but the plain Siamese architecture might not be enough owing to its structural simplicity and lack of motion information. This paper aims to propose a new MOT scheme to overcome the existing problems in the conventional MOTs. Feature pyramid Siamese network (FPSN) is proposed to address the structural simplicity. The FPSN is inspired by a feature pyramid network (FPN) and it extends the Siamese network by applying FPN to the plain Siamese architecture and by developing a new multi-level discriminative feature. A spatiotemporal motion feature is added to the FPSN to overcome the lack of motion information and to enhance the performance in MOT. Thus, FPSN-MOT considers not only the appearance feature but also motion information. Finally, FPSN-MOT is applied to the public MOT challenge benchmark problems and its performance is compared to that of the other state-of-the-art MOT methods.
BibTeX
@inproceedings{lee2018multiple,
  author = {Sangyun Lee and Euntai Kim},
  title = {{Multiple Object Tracking via Feature Pyramid Siamese Networks}},
  booktitle = {IEEE Access},
  year = {2018},
  doi = {10.1109/access.2018.2889442},
}
Figure