In multi-object tracking (MOT), the standard paradigm is "tracking-by-detection." An object detector identifies bounding boxes in each frame, and a data association algorithm links these boxes across time to form continuous trajectories. However, this pipeline faces a fundamental dilemma: where should we set the detection confidence threshold?
If we set the threshold too high, we filter out background noise, but we lose track of objects when they become partially occluded, blurred, or distant. The moment an object's detection score dips, its trajectory fragments. Conversely, if we lower the threshold to preserve these challenging targets, we introduce a massive influx of low-confidence false positives. In a traditional single-stage matching framework—such as applying the Hungarian algorithm globally across all detections—this noise causes severe identity switches, as spurious background boxes mistakenly "steal" the identities of true tracks.
Hierarchical data association resolves this dilemma by breaking the matching process into a sequential cascade. Instead of treating all detections and trajectories as a single homogeneous pool, it prioritizes certainties before resolving ambiguities.
The Core Mechanism: Lock in Certainties, Then Recover Targets
The intuition behind hierarchical association is elegant: by reducing the search space in stages, we drastically lower the probability of a false match.
Consider a basic two-stage confidence cascade:
- Stage 1 (High-Confidence Matching): The tracker isolates only the high-scoring detections and matches them to existing trajectories. Because both the detections and the active tracks are highly reliable, this step is clean and virtually error-free. It "locks in" the easy, unambiguous matches.
- Stage 2 (Low-Confidence Recovery): The tracker takes the remaining unmatched trajectories and attempts to link them to the low-scoring detections that were initially set aside.
[ All Detections ]
│
├─► High Confidence ──► [ Stage 1 Match ] ──► Update Active Tracks
│ │
│ (Unmatched Tracks)
│ ▼
└─► Low Confidence ──► [ Stage 2 Match ] ──► Recover Occluded/Blurred Tracks
│
(Remaining Noise) ──► Discard
Why does this two-stage approach prevent low-confidence noise from ruining the tracking quality? In Stage 2, the pool of available trajectories has been heavily depleted because the clear targets were already paired off. With fewer active trajectories left to compete for matches, a low-confidence false positive has a much lower mathematical chance of accidentally aligning with a valid track. This allows the system to safely recover occluded or blurred objects without polluting the clean trajectories.
This strategy was powerfully demonstrated by Yifu Zhang et al. in their benchmark-setting framework, ByteTrack: Multi-object Tracking by Associating Every Detection Box. They showed that low-score boxes are highly valuable for maintaining trajectory continuity through periods of blur or obstruction, provided they are processed conditionally. Similarly, Ziqi Pang in SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking adapted this logic to 3D LiDAR tracking, validating low-score detections against motion predictions in a secondary stage to prevent the premature termination of tracks caused by sensor noise.
Dimensions of the Hierarchy
The principle of hierarchical prioritization is highly adaptable and manifests across several distinct dimensions of the tracking pipeline:
1. Track-Level Confidence
Instead of filtering incoming detections, trackers can filter existing trajectories based on their historical reliability. Early work by Kuk-Jin Yoon et al. in Robust Online Multi-object Tracking Based on Tracklet Confidence and Online Discriminative Appearance Learning categorized trajectories into high- and low-confidence groups based on track length and occlusion history. Stable, long-lived tracks are extended using local frame-by-frame matching, while fragmented or lost tracks trigger a wider global re-identification search. Honghong Yang et al. in Multi-Pedestrian Tracking Based on Improved Two Step Data Association similarly partitioned trajectories to ensure that established, high-confidence paths lock in their detections first, preventing short-lived or noisy tracklets from causing identity switches. Even in detection-free paradigms, such as the point-trajectory clustering proposed by Katerina Fragkiadaki et al. in Detection free tracking: Exploiting motion and topology for segmenting and tracking under entanglement, a two-stage strategy is used: long-lived trajectories first establish a stable scene skeleton, and shorter, volatile trajectories are integrated later.
2. Spatial and Geometric Scaling
Trackers can also hierarchically relax their spatial constraints. For objects with highly irregular motions, matching within a tight spatial window is reliable for constant velocities but fails during sudden acceleration. Fan Yang et al. solved this in Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space by cascading the matching space. Their method uses a restrictive spatial buffer for predictable trajectories in the first stage, followed by a secondary stage with an expanded spatial buffer to capture objects undergoing sudden maneuvers or recovering from occlusion. In agricultural drone tracking, Danny Xie-Li et al. in PineSORT: A Simple Online Real-time Tracking Framework for Drone Videos in Agriculture extended this to three distinct stages, sequencing high-confidence Intersection-over-Union (IoU) matching, low-confidence recovery, and finally distance-based geometric association to handle objects near frame boundaries.
3. Feature and Modality Cascades
When tracking in complex environments, systems often cascade from strict multi-modal matching to simpler geometric fallbacks. For instance, in SportMamba: Adaptive Non-Linear Multi-Object Tracking with State Space Models for Team Sports, the authors prioritize high-confidence detections using a rich combination of motion and appearance cues, but relegate their second stage to exclusive spatial constraints to recover tracklets obscured by motion blur. For extreme crowding, Yunqi Zhao et al. in DynamicTrack: Advancing Gigapixel Tracking in Crowded Scenes employ a tiered approach that prioritizes joint head-body detections before falling back to individual part matches when full bodies are occluded.
Generalizing to Multi-Level and Hybrid Architectures
The success of the two-stage cascade has led researchers to generalize this principle into multi-level and hybrid architectures. Rather than relying on a binary split (high vs. low confidence), Cheng-Yen Yang et al. in Multi-Object Tracking by Iteratively Associating Detections with Uniform Appearance for Trawl-Based Fishing Bycatch Monitoring partitioned detection scores into multiple arbitrary confidence levels. By iteratively matching tracks against progressively lower certainty levels, they demonstrated robust identity continuity in highly chaotic, low-visibility industrial environments.
Furthermore, hierarchical association serves as a computational gatekeeper. Resolving simple tracking cases early allows systems to reserve intensive processing for truly ambiguous scenarios. This design pattern spans a wide chronological range of tracking research:
- Early Geometric Cues: Horst Possegger et al. in Occlusion Geodesics for Online Multi-object Tracking used standard bipartite matching for visible detections first, reserving complex geodesic pathfinding through known occlusion zones only for lost targets.
- Feature Optimization: Fengwei Yu et al. in POI: Multiple Object Tracking with High Performance Detection and Appearance Feature and Daniel Stadler et al. in A Strong Baseline for Multi-Person Tracking in Thermal Infrared Imagery structured pipelines to lock in strong appearance and spatial matches first, preventing noisy detections from corrupting deep feature representations.
- Modern Transformers: In TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking, Peng Chu et al. designed a hierarchical pipeline that resolves high-confidence matches using simple, lightweight motion models before employing a heavy spatial-temporal graph transformer to untangle complex, ambiguous associations.
Whether implemented via simple threshold rules or deep neural networks, hierarchical data association works because it respects the underlying certainty of the data. By solving the easiest parts of the puzzle first, it transforms a highly noisy global optimization problem into a series of clean, manageable sub-tasks.
Key Foundational Work
Robust Online Multi-object Tracking Based on Tracklet Confidence and Online Discriminative Appearance Learning (2014) — contributed 2 concept elements to this direction.
Occlusion Geodesics for Online Multi-object Tracking (2014) — contributed 1 concept elements to this direction.
Detection free tracking: Exploiting motion and topology for segmenting and tracking under entanglement (2011) — contributed 1 concept elements to this direction.