Visual object tracking is fundamentally an online learning problem. Given only a single bounding box in the initial frame, a tracker must locate the target across a sequence of thousands of frames. As the target moves, it undergoes drastic appearance changes caused by deformations, out-of-plane rotations, and lighting shifts.
To handle these variations, a tracker must be plastic: it must continuously update its internal model using newly observed frames. However, this plasticity introduces a dangerous positive feedback loop. If a tracker updates its model using a frame where the target is partially occluded or blurred, it inadvertently incorporates background noise or distractor features into its target representation. In subsequent frames, the corrupted model is even more likely to target the background, leading to an irreversible failure mode known as model drift.
This tension is a classic manifestation of the stability-plasticity dilemma. To prevent drift, modern tracking frameworks decouple their representations into multiple temporal scales, gate their updates based on confidence, and mathematically anchor their models to the original target identity.
Decoupling Timescales via Dual-Path Memory
The most intuitive way to break the self-reinforcing error loop of model drift is to avoid relying on a single, monolithic appearance model. Instead, architectures can split the tracking process into parallel paths operating at different timescales: a highly stable path anchored to the target's original identity, and a highly plastic path that adapts to immediate changes.
┌───────────────┐
│ Initial Frame │
└───────┬───────┘
│
▼
┌─────────────────────┐
│ Long-Term Anchor │ (Stable, Never Drifts)
└──────────┬──────────┘
│
▼
[Current Frame] ──► [Matcher / Fusion] ◄── [Short-Term Model] (Plastic, Captures Deformations)
▲
│ (Conditional Update)
┌───────┴───────┐
│ Recent Frames │
└───────────────┘
In Clustering of static-adaptive correspondences for deformable object tracking, Nebehay et al. demonstrated this by combining "static" correspondences (derived from the uncorrupted first frame) with "adaptive" correspondences (learned from recent frames). If the object deforms, the adaptive points maintain the track; if the adaptive points begin drifting onto the background, the static points pull the tracker back to reality.
This multi-timescale intuition has been scaled up to deep architectures and dictionary learning:
- Multi-Lifespan Dictionaries: Xing et al. structured template updates as an online incremental dictionary learning problem, maintaining short-, middle-, and long-term templates simultaneously to capture both transient and permanent target traits.
- Dual-Template Co-Evolution: In drone tracking scenarios where targets undergo extreme perspective shifts, Zhang et al. developed a framework that co-evolves a static initial template and a dynamic template to suppress background clutter.
- Asymmetric Correlation Filters: Ma et al. deployed an aggressive short-term correlation filter for precise frame-to-frame localization alongside a conservative long-term filter that acts as a validator to trigger re-detection during total occlusions.
- Dual-Path Transformers: For multi-object tracking, Kim et al. designed a dual-path temporal decoder that explicitly separates the network's focus, dedicating one path to tracking rapid appearance changes and a parallel path to strictly preserving long-term identity stability.
By separating these responsibilities, a temporary occlusion might corrupt the short-term memory path, but the long-term path remains unpoisoned, allowing the tracker to recover once the target emerges from behind an obstacle.
Gated Fusion and Conditional Plasticity
Simply maintaining two representations is not enough; the tracker must also intelligently fuse them and decide when to allow updates to the plastic component. If the system updates blindly on every frame, noise will eventually overwhelm the model.
To solve this, the authors of Deformable Object Tracking With Gated Fusion introduced a soft-gate fusion mechanism. When an object undergoes ordinary motion, the gate prioritizes stable, rigid features. When a complex deformation occurs, the gate dynamically shifts weight toward a deformable convolution layer, allowing the network to adapt its sampling grid without sacrificing baseline stability.
Another approach relies on active verification or "oracles" to break the self-learning loop. In Efficient Version-Space Reduction for Visual Tracking, Oba et al. utilized an active learning ensemble where a committee of short-term classifiers queries a long-term oracle model only when sample uncertainty crosses a specific threshold. Similarly, Burceanu et al. designed a "Society of Tracking Parts" that restricts deep network updates exclusively to "Highly Confident Frames," ensuring that the model never learns from ambiguous or occluded states.
Taking this concept to its logical conclusion, Kiran proposed a generative approach using an adversarial auto-encoder to predict future target appearances, paired with an explicit change detector that completely pauses template updates the moment an occlusion is suspected.
Mathematical Anchors Against Drift
Beyond architectural splits and gating heuristics, the stability-plasticity dilemma can be managed directly within the optimization and update equations of the tracker.
A common heuristic for updating online models is the Exponential Moving Average (EMA), which updates the model at time $t$ using a constant historical weight $\alpha$:
$$M_t = (1 - \alpha) M_{t-1} + \alpha f_t$$
The flaw in standard EMA is that it possesses no memory of the original target; a long occlusion can completely wash away the initial ground-truth representation. To fix this mathematically, Huang et al. formulated Re2EMA, which regularizes the update step and introduces an explicit reinitialization term. This term acts as a mathematical anchor, constantly pulling the running model back toward the uncorrupted initial frame $M_0$:
$$M_t = \arg\min_{M} \left( \|M - M_{t-1}\|^2 + \lambda_1 \|M - f_t\|^2 + \lambda_2 \|M - M_0\|^2 \right)$$
By embedding the initial target appearance ($M_0$) directly into the loss function of the update step, the optimization ensures that no matter how plastic the model must become to track a deformation ($f_t$), it can never drift entirely away from the true identity of the object.
Key Foundational Work
Efficient Visual Object Tracking with Online Nearest Neighbor Classifier (2010) — contributed 1 concept elements to this direction.
Robust Object Tracking with Online Multi-lifespan Dictionary Learning (2013) — contributed 1 concept elements to this direction.
Pose-Configurable Generic Tracking of Elongated Objects (2013) — contributed 1 concept elements to this direction.