Tracking Objects Across Large Temporal Gaps

179 concepts133 papers1 proposal

Most modern multi-object tracking (MOT) algorithms rely on a comfortable assumption: the world moves smoothly, and cameras capture it fast enough that an object in one frame will heavily overlap with its position in the next. This spatial continuity allows simple geometric metrics, like Intersection-over-Union (IoU), to effortlessly link object detections across time.

However, this assumption completely shatters in low-frame-rate (LFR) scenarios—such as wide-area aerial surveillance operating at 1–2 Hz, power-saving edge devices, or high-speed tracking environments. When the time interval between frames is large, objects travel distances far exceeding their own dimensions. This creates a fundamental challenge: the spatial overlap constraint drops to zero, causing traditional tracking pipelines to fail entirely.

The Binary Failure of Spatial Overlap

To understand why low frame rates degrade tracking, consider the geometry of a bounding box association. In a high-frame-rate video, a vehicle moving down a street shifts by only a fraction of its width per frame. Its bounding boxes at time $t$ and $t+1$ intersect significantly, yielding a high IoU score that uniquely identifies it.

As the sampling frequency drops, the physical distance traveled between observations increases. As noted by Kim et al. in EagerMOT: 3D Multi-Object Tracking via Sensor Fusion, geometric metrics like 3D IoU are highly sensitive to sensor sampling rates. When an object moves faster or further than its own volume within a single interval, the intersection between consecutive bounding boxes drops to exactly zero.

High Frame Rate:  [ Box t ][ Box t+1 ]  --> High IoU (Success)
Low Frame Rate:   [ Box t ] . . . . . . . . . [ Box t+1 ] --> Zero IoU (Failure)

This creates a binary failure mode. A similarity score of zero provides no gradient, no direction, and no structural information to tell an association algorithm which box belongs to which trajectory. As Rozumnyi et al. observe in The World of Fast Moving Objects, this breakdown of temporal continuity forces a paradigm shift away from local appearance or proximity matching toward methods that explicitly accommodate large spatial displacements.

Buffering and Scaling the Matching Space

The most direct way to repair a broken geometric metric is to artificially restore the overlap. If the physical detections do not intersect, we can expand the mathematical space around them until they do.

Yang et al. tackle this problem in Hard to Track Objects with Irregular Motions and Similar Appearances? Make It Easier by Buffering the Matching Space by introducing a "buffer zone" that inflates bounding boxes during the matching phase. By expanding the boundaries, the tracker ensures that even when an object undergoes a large, non-incremental displacement, its buffered box still intersects with the next detection, generating a non-zero similarity score that can guide data association.

This concept is particularly vital when tracking small, high-precision features. In A Distributed Tracking Algorithm for Counting People in Video by Head Detection, Kuplyakov et al. demonstrate that high-precision detectors (like those focusing solely on human heads) suffer from zero-overlap failures even with slight movements. The authors show that artificially scaling bounding boxes during the association phase acts as a vital buffer, compensating for sparse temporal sampling and preventing frequent identity swaps.

Shifting from Local Steps to Global Trajectories

Standard tracking frameworks are typically Markovian: they assume that the best way to find an object at time $t+1$ is to look strictly at its state at time $t$. When temporal resolution is low, this step-by-step approach accumulates errors rapidly.

To bypass this limitation, researchers look beyond immediate frame-to-frame pairs toward global constraints. In Non-Markovian Globally Consistent Multi-object Tracking, Maksai et al. argue that local motion cues become entirely unreliable when temporal gaps are wide. Instead, the authors enforce global path-based constraints across the entire video sequence. By optimizing the entire trajectory at once and leveraging typical scene behaviors, the system can bridge large temporal gaps where local continuity is lost.

Taking this behavioral intuition further, Yamaguchi et al. demonstrate in Who are you with and where are you going? that tracking accuracy in sporadic settings improves significantly when targets are modeled as decision-making agents. By estimating long-term hidden variables—such as a pedestrian's intended destination or social group affiliations—the tracker can predict where an object will reappear after a long temporal gap, bypassing the need for continuous visual or spatial tracking.

Observation-Centric Motion Modeling

When spatial overlap vanishes, trackers often fall back on motion models (like the Kalman filter) to predict an object's future position. However, standard motion models assume linear updates with small, high-frequency noise. Over large temporal gaps, the prediction error compounds exponentially, causing the filter's search window to drift far from the true target.

To solve this compounding error, Pang et al. introduce an "observation-centric" philosophy in Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking. Instead of allowing the motion model to dictate tracking decisions blindly during periods of low visibility or low frame rates, their framework uses subsequent real observations to retrospectively correct the accumulated errors of the Kalman filter. This prevents the motion model from diverging during long intervals.

For generic objects that exhibit highly non-linear or irregular movements, Ogawa proposes a multi-state motion framework in FRoG-MOT: Fast and Robust Generic Multiple-Object Tracking by IoU and Motion-State Associations. By tracking multiple independent motion states (such as velocity, acceleration, and shape changes) and adaptively discarding outlier states during association, the author demonstrates that robust motion modeling can completely replace spatial overlap metrics as the primary mechanism for maintaining track identity when video frame rates drop.

Key Foundational Work

Monocular tracking of the human arm in 3D: real-time implementation and experiments (1996) — contributed 4 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 127 papers
  2. Version 1 · 2026-10-04 · 113 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_low_frame_rate_tracking_degradation_d7d1cbad,
  title = {low-frame-rate tracking degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-frame-rate-tracking-degradation/rdr_d7d1cbad41a74674836cddfe417b7a37},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_low_frame_rate_tracking_degradation_d7d1cbad_v2,
  title = {low-frame-rate tracking degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/low-frame-rate-tracking-degradation/rdr_d7d1cbad41a74674836cddfe417b7a37/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake