Abstract
FAMNet is an end-to-end deep learning framework for Multiple Object Tracking (MOT) that jointly optimizes feature extraction, affinity estimation, and multi-dimensional data association. By making the assignment algorithm itself differentiable through a Power Iteration Layer, the model can learn to track objects by backpropagating errors directly from final tracking results to raw image pixels. Evaluated on major benchmarks including MOT2015, MOT2017, and KITTI, FAMNet achieves state-of-the-art or highly competitive performance, demonstrating the significant advantages of joint optimization over disjointed tracking-by-detection pipelines.
Introduction
Most modern Multiple Object Tracking (MOT) systems follow a "tracking-by-detection" philosophy. This process is typically composed of three separate modules: feature extraction, affinity metric calculation, and data association (often using discrete optimization like the Hungarian algorithm). Traditionally, these modules are optimized in isolation, which prevents the feature extractor from being "task-aware" regarding the association logic. Furthermore, the non-differentiable nature of standard association algorithms creates a bottleneck for end-to-end training. FAMNet addresses these limitations by integrating all three components into a single, differentiable neural network.
Methodology
FAMNet introduces a unified architecture that allows for end-to-end gradient flow from the tracking loss to the input images.
The framework consists of several innovative sub-networks:
- Feature and Affinity Sub-Networks: Utilizing a Siamese-style architecture, these layers extract features from multiple frames simultaneously. By incorporating a "Detection Attention Mask" and cross-correlation, the network fuses appearance and motion clues into a higher-order affinity tensor.
- MDA Sub-Network: Multi-Dimensional Assignment (MDA) is critical for maintaining global consistency across frames. FAMNet transforms this traditionally discrete optimization problem into a differentiable Power Iteration Layer. By approximating the assignment solution through iterative matrix operations, the model enables backpropagation through the data association step.
- SOT Integration: To handle false negatives from external detectors, FAMNet integrates Single Object Tracking (SOT) predictions as "virtual candidates." This ensures the tracker maintains target continuity even when detections are missing.
Experiments and Results
The model was rigorously evaluated on four major MOT benchmarks: MOT2015, MOT2017, KITTI-Car, and UA-DETRAC.
Key findings from the experiments include:
- State-of-the-Art Accuracy: On the MOT2017 benchmark, FAMNet achieved a MOTA of 52.0, outperforming several established online and offline trackers. It also reached a MOTA of 40.6 on MOT2015 and 77.1 on KITTI.
- Impact of Joint Learning: An ablation study revealed that end-to-end fine-tuning of the entire pipeline provided a ~10% boost in MOTA compared to using pre-trained modules without joint optimization. This confirms that task-specific feature learning is vital for robust tracking.
Conclusion
FAMNet represents a significant step forward in MOT by proving that the entire tracking pipeline, including the complex data association step, can be learned end-to-end. While the current implementation runs at approximately 0.6 FPS—limiting its use in real-time applications—the framework establishes a powerful foundation for future research into differentiable optimization for computer vision tasks. Future work will focus on improving computational efficiency and exploring more complex motion models.
Summary updated