In visual object tracking, a system is given a single bounding box in the first frame of a video and tasked with following that object indefinitely. This sounds straightforward, but it hides a fundamental paradox known as the stability-plasticity dilemma.
If a tracker is too "stable"—meaning it only looks for the object exactly as it appeared in the first frame—it will fail the moment the object rotates, moves into a shadow, or changes shape. However, if the tracker is too "plastic"—meaning it rapidly updates its internal model based on the most recent frames—it risks "drifting." Drift occurs when the tracker accidentally incorporates background clutter or an occluding object into its model, eventually losing the target entirely.
The Mechanism of Model Drift
The core of the problem is what Ran Tao, Deepak Gupta, and A.W.M. Smeulders call "model decay" in Model Decay in Long-Term Tracking. When a tracker makes a slightly imperfect prediction, and then uses that prediction to update its internal template, it "poisons" its own memory. Over dozens of frames, these tiny errors accumulate until the tracker's internal representation of the object looks more like the background than the original target.
As Nicholas D. H. Dowson and Richard Bowden noted in their work on Simultaneous Modeling and Tracking (SMAT) of Feature Sets, a tracker must balance the need for template stability against the necessity of adapting to changes in lighting and pose.
Multi-Lifespan Memory and Reservoirs
One intuitive solution is to stop treating memory as a single, rolling update. Instead of just remembering the "last seen" version of an object, researchers have moved toward hierarchical memory structures.
In Robust Object Tracking with Online Multi-lifespan Dictionary Learning, Junliang Xing and colleagues proposed a "multi-lifespan" dictionary. This system maintains short-term, middle-term, and long-term templates simultaneously. If the short-term model becomes corrupted by a temporary occlusion, the long-term model acts as an anchor to pull the tracker back to reality. Similarly, Xi Li and Anthony Dick utilized "time-weighted reservoir sampling" in Online Metric-Weighted Linear Representations for Robust Visual Tracking to ensure the tracker retains a diverse history of appearances rather than just the most recent (and potentially noisy) ones.
Dual-Path Architectures: Anchors and Scouts
Another dominant strategy is to split the tracking architecture into two parallel paths: one that is fixed and one that is adaptive.
Georg Nebehay and Roman P. Pflugfelder pioneered this with Clustering of static-adaptive correspondences for deformable object tracking, which combines "static" correspondences from the initial frame with "adaptive" ones from recent frames. This concept was later modernized for deep learning by Lijun Wang and Huchuan Lu in Visual Tracking with Fully Convolutional Networks. Their FCNT tracker uses a dual-track update mechanism where high-level semantic features remain static to preserve the object's identity, while low-level discriminative features adapt to capture immediate changes in appearance.
More recently, the Dual-Path Temporal Decoder by Hyunseop Kim and Yeong Jun Koh applied this logic to transformers, using one path to adapt to appearance and another to strictly preserve identity through complex motions.
Learning to Update
The most recent evolution in this field moves away from hand-crafted update rules (like the popular Exponential Moving Average) and toward "meta-learning." Instead of a human deciding how much to weight the new frame, a neural network is trained to make that decision.
In Learning to Update for Object Tracking, Bi Li 0005 and Wenjun Zeng introduced an RNN that learns the optimal update strategy from large-scale video datasets. This allows the tracker to recognize when a change in appearance is a legitimate deformation (requiring an update) versus a temporary occlusion (requiring the model to stay frozen).
Similarly, Jianglei Huang and Wengang Zhou 0001 proposed Re2EMA, which adds a "reinitialization" term to the update math. This term acts as a mathematical anchor, constantly pulling the adaptive model back toward the ground-truth appearance of the first frame to prevent the slow, creeping poison of model drift.