Mitigating Trajectory Fragmentation under Occlusion

319 concepts283 papers1 proposal

In Multi-Object Tracking (MOT), a system must assign unique, persistent identifiers to targets as they move through a scene. While modern object detectors are highly proficient at locating objects in individual frames, maintaining these identities over time remains a fragile process.

The primary catalyst for tracking failure is occlusion. When a target passes behind an obstacle, a larger vehicle, or another person, the visual evidence vanishes. When the target emerges frames later, the tracker faces a fundamental ambiguity: Is this a brand-new object, or is it a previously tracked entity re-emerging?

If the tracking algorithm fails to resolve this ambiguity, it suffers from trajectory fragmentation (breaking a single continuous path into disconnected segments) or an identity switch (wrongfully swapping the labels of two nearby objects). Understanding how modern tracking frameworks prevent these failures requires looking at how they bridge these temporal blind spots.


The Vulnerability of Local Tracking

To understand why trajectories fragment, we must look at the classic "tracking-by-detection" paradigm. Traditional online trackers operate greedily frame-by-frame. They use a motion model—typically a Kalman filter—to predict where an object will be in the next frame based on its velocity, and then use bipartite graph matching (like the Hungarian algorithm) to pair these predictions with new detections.

This approach works beautifully when frame rates are high and motion is linear. However, as noted by Patil et al., standard filtering techniques struggle to maintain a persistent representation when an object is temporarily obscured. During an occlusion:

  1. The detection disappears: The tracker receives no new bounding boxes or point clouds to anchor the target.
  2. The motion model degrades: Without new measurements, the uncertainty of the motion model grows exponentially. If the target changes speed or turns while hidden, the prediction will drift significantly from reality.
  3. The track terminates: To avoid propagating false positives, online trackers typically delete a track if it remains undetected for more than a few frames.

As a result, when the object reappears, the system treats it as a completely new detection, assigning it a new ID and leaving behind a fragmented trajectory history.


Re-Identification: Memorizing Visual Signatures

The most direct solution to trajectory fragmentation is to give the tracker a "visual memory" that persists even when spatial tracking fails. Instead of relying solely on where an object should be, trackers extract deep appearance embeddings to capture what the object looks like.

In their seminal work on Deep SORT, Bewley et al. demonstrated that integrating a deep learning-based appearance metric allows trackers to maintain identities through much longer periods of occlusion, reducing identity switches by nearly half. When a target disappears, its appearance embedding is stored in a gallery. Upon re-emergence, the system computes the cosine distance between the new detection's embedding and the historical gallery, allowing it to successfully reconnect the track even if the spatial prediction has completely drifted.

[Frame t] Target Visible  --> Extract Appearance Embedding --> Save to Gallery
[Frame t+1] Target Occluded --> Spatial Prediction Drifts    --> Track Suspended
[Frame t+n] Target Reappears--> Match Embedding to Gallery  --> Identity Restored

To optimize this process, Li et al. proposed a two-stage feature learning paradigm. They separated feature extraction into Single-Shot learning for immediate frame-to-frame association and Multi-Shot learning specifically optimized for re-identifying targets after long-term occlusions. This dual-rate approach ensures that short-term localization remains highly precise without sacrificing the robust, invariant features needed to bridge extended visual gaps.


Global Optimization: Stitching Tracklets Retrospectively

While deep appearance features help, they can fail in crowded environments where targets look identical—such as a crowd of pedestrians wearing similar clothing or a fleet of identical vehicles. To resolve these highly ambiguous scenarios, researchers often abandon strict real-time, frame-by-frame tracking in favor of near-online or offline global optimization.

Instead of making irreversible decisions at every frame, these methods group continuous, unambiguous frame sequences into short segments called tracklets. When an occlusion occurs, the tracklet terminates. The tracking problem is then reframed as a global optimization task: What is the most mathematically consistent way to link these isolated tracklets across time?

Choudhuri et al. addressed this by formulating the tracking problem within an assignment space, using dynamic programming to discover the optimal sequence of object-to-object links across an entire video sequence. Similarly, Li 0001 et al. introduced end-to-end Graph Networks to model the global relationships between targets. By treating tracklets as nodes and potential temporal links as edges, the network can look forward and backward in time to resolve complex inter-occlusion events that would cause a greedy frame-by-frame tracker to fail.


Object Permanence and Contextual Priors

The most sophisticated strategy for maintaining identity continuity mimics human cognitive development: object permanence, the understanding that objects continue to exist even when they cannot be seen.

Instead of allowing a track to simply vanish during an occlusion, advanced frameworks use environmental and behavioral context to maintain a "ghost" state of the object. For example, Caesar et al. utilized vectorized lane maps as a spatial prior in autonomous driving scenarios. If a vehicle disappears behind a large truck, the system does not assume it has ceased to exist; instead, it constrains the vehicle's hidden trajectory to the geometry of the lane map, accurately "filling in the blanks" until it reappears.

In a similar vein, Pellegrini et al. introduced social-aware motion models. By modeling how humans proactively navigate crowds to avoid collisions, their system predicts highly realistic, non-linear paths during periods of total occlusion. By constraining the search space using physics and social expectations, the tracker drastically reduces the likelihood of identity switches when the target finally re-enters the sensor's field of view.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_trajectory_fragmentation_96d63692,
  title = {trajectory fragmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/trajectory-fragmentation/rdr_96d63692168647d48e10f9eb191a56ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_trajectory_fragmentation_96d63692_v1,
  title = {trajectory fragmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/trajectory-fragmentation/rdr_96d63692168647d48e10f9eb191a56ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake