For years, computer vision treated multi-object tracking (MOT) as a two-step assembly line. First, an object detector located targets within independent frames. Second, a data association algorithm linked these discrete bounding boxes across time. This "tracking-by-detection" paradigm was simple, but fundamentally fragile. If a detector missed a target due to motion blur, unusual poses, or temporary occlusion, the downstream tracker broke, causing identity switches or fragmented trajectories. As noted by Nuno Vasconcelos et al. in Saliency-based discriminant tracking, the field historically lacked a unified mathematical framework capable of seamlessly transitioning from initial object discovery to continuous tracking.
To resolve this fragility, modern architectures merge detection and tracking into a single, end-to-end differentiable network. By optimizing both tasks simultaneously, these frameworks achieve mutual reinforcement: tracking history helps the detector resolve spatial ambiguities, while detection features are co-optimized to satisfy temporal consistency.
The Feedback Loop: How Tracking Informs Detection
Why should an object detector operate blindly on each frame when it has access to the immediate past? In a decoupled pipeline, errors propagate strictly forward. If the detector fails, the tracker has no data to work with. In a unified framework, however, tracking data acts as a temporal prior that actively guides the detection process.
An early realization of this concept appeared in the work of Chi Su in PKU-NEC @TRECVID2011 SED: Sequence-Based Event Detection in Surveillance Video, which demonstrated that a recursive state estimator could constrain a detector's search space to prevent error propagation. Similarly, Kai Jüngling et al. in Detection and tracking of objects with direct integration of perception and expectation injected predicted track states directly into the detection voting space, treating expectation as a feature-level input.
In modern deep learning architectures, this principle manifests as explicit architectural feedback loops. For example, the framework proposed by Junho Koh et al. in Joint 3D Object Detection and Tracking Using Spatio-Temporal Representation of Camera Image and LiDAR Point Clouds feeds tracking data back into the detection stage. This allows the detector to leverage temporal context and memory of prior object locations, drastically reducing false negatives in challenging environmental conditions.
Traditional Pipeline:
[ Image ] ──> [ Independent Detector ] ──> [ Discrete Detections ] ──> [ Data Association ]
Unified Pipeline with Feedback:
[ Image ] ──> [ Co-Optimized Detector ] ──> [ Differentiable Association ]
▲ │
└─────────────────── [ Temporal Track Memory ] ──────┘
This memory mechanism is highly effective in crowded or chaotic scenes. Hantao Yao et al. developed this further in UTM: A Unified Multiple Object Tracking Model with Identity-Aware Feature Enhancement by introducing an Identity-Aware Feature Enhancement module. This module uses identity knowledge from previous tracklets to actively enhance current frame features, resolving detection ambiguities during heavy occlusion. When targets undergo rapid motion or severe deformation—such as the egocentric dog tracking explored by Mingzhen Huang et al. in Tracking Multiple Deformable Objects in Egocentric Videos—allowing the association logic to inform the detection process ensures continuous identity tracks and fewer dropped targets.
Co-Optimizing Representations for Localization and Association
In a decoupled pipeline, the feature extractor is typically trained solely on static detection losses (such as bounding box regression and classification). These features might excel at finding a generic object boundary, but they are often sub-optimal for distinguishing between two highly similar targets over time.
Joint architectures solve this by forcing a single backbone network to learn features that are simultaneously discriminative for spatial localization and robust for temporal association. As argued by Davi Frossard et al. in End-to-end Learning of Multi-sensor 3D Tracking by Detection, joint training prevents the sub-optimal feature alignment typical of disjoint pipelines, ensuring that detection scoring and matching metrics are co-optimized to minimize trajectory fragmentation.
A clear example of this synergy is found in SiamMOT: Siamese Multi-Object Tracking, where Bing Shuai et al. coupled a detection backbone with a region-based Siamese tracking branch. Because the network is trained end-to-end, the shared feature representations are explicitly tuned to be robust to both spatial deformation and temporal appearance changes.
This shared representation yields two major benefits:
- Computational Efficiency: It eliminates redundant feature extraction across separate models, as shown by Junbo Yin et al. in A Unified Object Motion and Affinity Model for Online Multi-Object Tracking.
- Feature Refinement: It allows downstream tracking losses to directly refine the low-level perception features. For instance, Aldrich Zhijun Pan et al. demonstrated in RaTrack: Moving Object Detection and Tracking with 4D Radar Point Cloud that jointly optimizing motion segmentation, scene flow, and data association allows tracking errors to backpropagate and improve the initial segmentation features.
Eliminating Heuristics via Differentiable Association
Traditional tracking relies heavily on discrete, non-differentiable post-processing steps. Algorithms like Non-Maximum Suppression (NMS) and the Hungarian matching algorithm operate on discrete sets of bounding boxes. Because you cannot compute a gradient through a discrete matching step, tracking errors cannot easily guide the upstream detector.
To achieve true end-to-end optimization, researchers have re-engineered data association to be fully differentiable. One approach, introduced by Yihong Xu et al. in How to Train Your Deep Multi-Object Tracker, replaces the discrete Hungarian algorithm with a differentiable "Deep Hungarian Net" (DHN). This allows the network to optimize parameters directly against proxies for standard tracking metrics like MOTA (Multi-Object Tracking Accuracy).
Another prominent strategy completely bypasses explicit matching and NMS by using track queries or cross-frame correlation. In these architectures, objects are tracked by propagating continuous embeddings across frames.
Differentiable Query Propagation:
[ Frame t-1 Track Embeddings ] ──┐
▼
[ Frame t Feature Map ] ──> [ Cross-Attention ] ──> [ Updated Tracks & New Detections ]
For example, Zhenwei Yang et al. in LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation and Tianyuan Zhang et al. in MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries utilize persistent track queries alongside detection queries. Instead of pairing boxes after the fact, the model updates existing track queries using cross-attention over the new frame's feature maps.
Similarly, Xuewu Lin et al. show in End-to-End Joint Multi-Object Detection and Tracking for Intelligent Transportation Systems that treating tracking as a cross-frame correlation problem allows a single network to handle detection and association implicitly. By propagating embeddings directly across frames via attention mechanisms, as explored by Brian Cheong et al. in JDT3D: Addressing the Gaps in LiDAR-Based Tracking-by-Attention, or by learning track management directly from data peaks, as shown by Chenxu Luo et al. in Exploring Simple 3D Multi-Object Tracking for Autonomous Driving, the network learns to initiate, maintain, and terminate tracks natively within a single forward pass.
Extending the Horizon to Trajectory Forecasting
Once detection and tracking are unified into a single spatio-temporal representation, the network naturally possesses the contextual information required to predict where an object will go next. Instead of stacking a forecasting module on top of a completed tracking pipeline, modern architectures perform detection, tracking, and trajectory forecasting simultaneously.
In Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data, Sergio Casas et al. integrated object detection and motion forecasting using Graph Neural Networks. This ensures that the detection backbone learns features optimized for future trajectory prediction while accounting for social interactions between targets.
By executing tracking and prediction in parallel, as proposed by Xinshuo Weng et al. in PTP: Parallelized Tracking and Prediction With Graph Neural Networks and Diversity Sampling, or by forecasting future optical flow maps in a single stage, as demonstrated by Yu-Wen Chen et al. in S2F2: Single-Stage Flow Forecasting for Future Multiple Trajectories Prediction, these systems eliminate multi-stage error propagation entirely. The result is a holistic perception-to-prediction pipeline where features are co-optimized for localization, identity consistency, and future intent estimation.
Key Foundational Work
Guest Editors' Introduction to the Special Section on CVPR Papers (2008) — contributed 1 concept elements to this direction.
A system for automatic face obscuration for privacy purposes (2006) — contributed 1 concept elements to this direction.