Object Permanence for Robust Multi-Object Tracking

This direction is split into other directions: trajectory fragmentation. Its last version stays readable and citable here.

Version 1 of 1382 papers424 conceptsfrozen 2026-10-04

In the classic "tracking-by-detection" paradigm, a computer vision system is only as good as its latest observation. If a car passes behind a tree or a pedestrian is momentarily eclipsed by a bus, the tracker often "forgets" the object existed. When the object reappears, the system treats it as a brand-new entity, leading to fragmented trajectories and identity switches.

To solve this, researchers are moving toward object permanence: the ability of a model to reason about an object’s existence and trajectory even when it is completely hidden from view.

The Identity Crisis of Occlusion

The fundamental challenge in Multi-Object Tracking (MOT) is maintaining a consistent ID across frames. In crowded environments—like the cornfields navigated by Zhong-Zhong Zhang’s TerraSentia robot or the surgical theaters described in ReMeDI-SAM3—occlusions are not rare edge cases; they are constant.

When visual evidence disappears, standard trackers lose their "anchor." Jie Li and colleagues argue in Learning to Track with Object Permanence that trackers must stop being purely reactive. Instead of waiting for a new detection, models should "hallucinate" the likely position of an occluded target using spatio-temporal memory. This shift from 2D image matching to 4D reasoning (3D space + time) allows the system to bridge gaps where the target is invisible.

Hallucinating the Hidden

One way to maintain identity is to "see" what isn't there. This is known as amodal perception—the cognitive ability to perceive the whole of a physical structure even when only a portion of it is visible.

  • Geometric Hallucination: In Seeing Behind Objects for 3D Multi-Object Tracking, Norman Müller proposes completing the 3D geometry of partially scanned objects. By predicting the full shape and canonical pose of a car or chair, the tracker creates a stable geometric signature that persists even as the visible pixels change or vanish.
  • Amodal Segmentation: Jasmin Breitenstein explores this in Foundation Models for Amodal Video Instance Segmentation, where the model predicts the full mask of a vehicle, including the parts hidden behind other cars. This "complete" mask provides a much more reliable target for temporal association than a fragmented, visible-only mask.

Bridging Gaps with Motion and Memory

If an object is gone for a long time, simple linear velocity models fail. Modern approaches use more sophisticated bridges:

  1. Spatio-Temporal Recurrence: Models like PermaTrack use recurrent neural networks (RNNs) to maintain an internal state of where objects should be. This memory allows the model to "track through" occlusions by maintaining a latent representation of the object's path.
  2. Flow-Based Splatting: Bishoy Galoaa and the team behind MOTE use optical flow and "softmax splatting" to project an object’s features into future frames. This effectively "carries" the identity forward through the occlusion until a new visual detection can confirm its location.
  3. Spatial Priors: In autonomous driving, Holger Caesar shows in Offline Tracking with Object Permanence that vectorized lane maps act as a powerful prior. If a car disappears, the system knows it is likely still following the lane, allowing it to "fill in the blanks" for auto-labeling.

Global Optimization over Local Matching

Rather than making a frame-by-frame decision, some researchers treat tracking as a global optimization problem. Paul Swoboda and Duy Minh Ho Nguyen utilize a "lifted multicut" formulation in LMGP.

Instead of just linking Frame A to Frame B, this approach looks at the entire video sequence as a graph. It uses 3D ground-plane projections to ensure that the links made between tracklets are physically plausible. Similarly, Siyu Tang demonstrated in A Multi-cut Formulation for Joint Segmentation and Tracking that low-level motion cues and high-level detections can be optimized together. If a detection is missing in the middle of a sequence, the "multicut" cost of breaking a long, consistent trajectory is higher than the cost of "hallucinating" the missing link.

By combining these mechanisms—amodal hallucination, recurrent memory, and global graph optimization—the field is moving toward a version of computer vision that doesn't just see the world, but understands the persistence of the objects within it.

Papers in this version

Showing 60 of 382 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_permanence_for_robust_multi_objec_c4c79577,
  title = {Object Permanence for Robust Multi-Object Tracking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-permanence-for-robust-multi-object-tracking/rdr_c4c79577982a481195e9531d5de24e25},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_permanence_for_robust_multi_objec_c4c79577_v1,
  title = {Object Permanence for Robust Multi-Object Tracking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-permanence-for-robust-multi-object-tracking/rdr_c4c79577982a481195e9531d5de24e25/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}