FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking

Peng Chu, Haibin Ling

Abstract

FAMNet is an end-to-end deep learning framework for Multiple Object Tracking (MOT) that jointly optimizes feature extraction, affinity estimation, and multi-dimensional data association. By making the assignment algorithm itself differentiable through a Power Iteration Layer, the model can learn to track objects by backpropagating errors directly from final tracking results to raw image pixels. Evaluated on major benchmarks including MOT2015, MOT2017, and KITTI, FAMNet achieves state-of-the-art or highly competitive performance, demonstrating the significant advantages of joint optimization over disjointed tracking-by-detection pipelines.

Introduction

Most modern Multiple Object Tracking (MOT) systems follow a "tracking-by-detection" philosophy. This process is typically composed of three separate modules: feature extraction, affinity metric calculation, and data association (often using discrete optimization like the Hungarian algorithm). Traditionally, these modules are optimized in isolation, which prevents the feature extractor from being "task-aware" regarding the association logic. Furthermore, the non-differentiable nature of standard association algorithms creates a bottleneck for end-to-end training. FAMNet addresses these limitations by integrating all three components into a single, differentiable neural network.

Methodology

FAMNet introduces a unified architecture that allows for end-to-end gradient flow from the tracking loss to the input images.

The FAMNet architecture integrates feature extraction, affinity estimation, and multi-dimensional assignment into a single differentiable pipeline.

The framework consists of several innovative sub-networks:

  • Feature and Affinity Sub-Networks: Utilizing a Siamese-style architecture, these layers extract features from multiple frames simultaneously. By incorporating a "Detection Attention Mask" and cross-correlation, the network fuses appearance and motion clues into a higher-order affinity tensor.
  • MDA Sub-Network: Multi-Dimensional Assignment (MDA) is critical for maintaining global consistency across frames. FAMNet transforms this traditionally discrete optimization problem into a differentiable Power Iteration Layer. By approximating the assignment solution through iterative matrix operations, the model enables backpropagation through the data association step.
  • SOT Integration: To handle false negatives from external detectors, FAMNet integrates Single Object Tracking (SOT) predictions as "virtual candidates." This ensures the tracker maintains target continuity even when detections are missing.

Experiments and Results

The model was rigorously evaluated on four major MOT benchmarks: MOT2015, MOT2017, KITTI-Car, and UA-DETRAC.

Performance comparison on the MOT2017 benchmark, where FAMNet (proposed) achieves a state-of-the-art MOTA of 52.0.

Key findings from the experiments include:

  • State-of-the-Art Accuracy: On the MOT2017 benchmark, FAMNet achieved a MOTA of 52.0, outperforming several established online and offline trackers. It also reached a MOTA of 40.6 on MOT2015 and 77.1 on KITTI.
  • Impact of Joint Learning: An ablation study revealed that end-to-end fine-tuning of the entire pipeline provided a ~10% boost in MOTA compared to using pre-trained modules without joint optimization. This confirms that task-specific feature learning is vital for robust tracking.

Conclusion

FAMNet represents a significant step forward in MOT by proving that the entire tracking pipeline, including the complex data association step, can be learned end-to-end. While the current implementation runs at approximately 0.6 FPS—limiting its use in real-time applications—the framework establishes a powerful foundation for future research into differentiable optimization for computer vision tasks. Future work will focus on improving computational efficiency and exploring more complex motion models.

Summary updated

Abstract

Data association-based multiple object tracking (MOT) involves multiple separated modules processed or optimized differently, which results in complex method design and requires non-trivial tuning of parameters. In this paper, we present an end-to-end model, named FAMNet, where Feature extraction, Affinity estimation and Multi-dimensional assignment are refined in a single network. All layers in FAMNet are designed differentiable thus can be optimized jointly to learn the discriminative features and higher-order affinity model for robust MOT, which is supervised by the loss directly from the assignment ground truth. We also integrate single object tracking technique and a dedicated target management scheme into the FAMNet-based tracking system to further recover false negatives and inhibit noisy target candidates generated by the external detector. The proposed method is evaluated on a diverse set of benchmarks including MOT2015, MOT2017, KITTI-Car and UA-DETRAC, and achieves promising performance on all of them in comparison with state-of-the-arts.
BibTeX
@inproceedings{chu2019famnet,
  author = {Peng Chu and Haibin Ling},
  title = {{FAMNet: Joint Learning of Feature, Affinity and Multi-dimensional Assignment for Online Multiple Object Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00627},
}
Figure