Late Corrections for Drifting Video Trackers

Many deployed vision systems pair a fast local model that runs every frame with a slow, accurate model that sits somewhere else: on an edge server, in a cloud queue, or on a specialized accelerator. The local model is a memory-based tracker, meaning a segmenter that stores its own past predictions and reads them back to label the next frame. It drifts, because every mistake it writes into memory becomes evidence for the next mistake. The remote model can fix the drift, but its answer arrives late. By the time a corrected mask for frame $t$ comes back over the network, the tracker is at frame $t+d$, and it has spent $d$ frames writing more errors into its memory. Today's systems either ignore this delay or respond to it bluntly. They wait, they reset the tracker to a correction that is already stale, or they redo all the intervening frames.

This note proposes a cheap way to apply a correction after the fact. The method rests on a structural fact about memory-based segmenters such as Space-Time Memory networks and XMem. Each stored memory entry has a key, computed from the image alone, and a value, computed from the image together with the predicted mask. Drift corrupts only the values. So when a late correction arrives, the keys cached during the delay window are still valid. The tracker can rebuild the corrupted values from the corrected anchor by re-running only its lightweight readout and decoder over the window, with no image re-encoding. It can also do this for only a subsample of the window frames, chosen by uncertainty. If this works, a delayed correction should recover most of the benefit of an instantaneous one, at a small fraction of the cost of full replay.

Core Claim

The claim has two parts. First, under realistic round-trip delays of one to several seconds, how a correction is applied when it arrives matters as much as when it is requested. Second, retroactive value rebuilding over cached keys closes most of the gap between "hard reset on arrival" and "correction with zero delay." The secondary claim is about scheduling. Once rectification is cheap, the best time to request a correction depends on both the tracker's own uncertainty and the expected delay, so requests should be sent earlier when delay is long.

The motivation is well documented. LVOS builds a long-video benchmark and shows two things: a drop of about 25% $\mathcal{J}\&\mathcal{F}$ relative to short clips, and a gain of up to 20% $\mathcal{J}\&\mathcal{F}$ from placing oracle masks in memory. That gain is the size of the prize. Space-Time Memory networks introduced the key/value memory bank whose recursive poisoning we target. They are also the architecture whose key/value split we exploit.

Relation to Prior Work

Three lines of work correct drift, and all of them assume the correction is instantaneous. Lazy-XMem uses masked entropy of the tracker's softmax output to trigger pseudo-corrections from SAM-HQ or human clicks, and injects the corrected masks immediately. CL-MC uses a state machine to hard-reset SAM2's memory from a co-located detector. Selector Net gates which masks enter memory, at about 2.2 s per frame. We reuse two of their ingredients: Lazy-XMem's entropy signal and CL-MC's notion of overwriting memory. What we change is the timing assumption. A correction refers to the past, and it lands on a memory that has moved on since the request was sent.

Anchor-based methods avoid recursion altogether. FEELVOS matches each frame to both frame 0 and frame $t-1$, and Anchor Diffusion propagates directly from a reference frame. These methods constrain our design. They show that a clean anchor is powerful, but a single fixed anchor goes stale on minute-long videos with large appearance changes. Late corrections can be read as fresh anchors that arrive at irregular, delayed times.

The latency side of the problem comes from LTED-Ada. It learns when to offload frames from a Raspberry Pi tracker to an edge Faster R-CNN, measuring about 0.47 s for local tracking against about 1.45 s for edge detection. When a detection returns, it simply re-initializes a bounding-box optical-flow tracker, so it has no persistent memory to repair. We take its latency model and its offloading question, and replace the re-initialization with memory repair at the pixel-mask level.

The quantum-classical hybrid literature in the file shows how extreme interface delays can become. Kölle et al. estimate that a 95-second classical training run becomes 10.9 hours on quantum hardware. Parameter recycling for VQE warm-starts each optimization from a related earlier problem, which cuts iterations by about 93%, but it never asks when recycled state has gone stale. That question is the analogue of ours. We do not build a quantum system. These papers motivate one experimental choice: a latency sweep with heavy tails reaching well beyond edge-server delays.

Research Program

The base tracker is XMem with frozen weights, because both Lazy-XMem and LVOS evaluate on it and its key encoder is mask-independent. Each memory entry is tagged with a timestamp, its cached key, its value, and the masked entropy of the prediction that produced it.

The corrector is SAM-HQ prompted from the tracker's box. Lazy-XMem's pseudo-correction setup serves as the reference. The corrector runs behind an emulated channel, and a correction requested at frame $t_s$ returns at frame $t_s+d$. The delay $d$ is drawn from one of three profiles: constant, a trace matching LTED-Ada's roughly 1.45 s edge latency (about 40 frames at 30 fps), and a heavy-tailed distribution standing in for congested queues.

When the correction for $t_s$ arrives, the rectifier makes three moves. It writes the corrected mask's value at $t_s$ as a high-trust entry. It then walks forward through a subset $S$ of the window $(t_s, t_s+d]$. For each frame in $S$, it re-runs memory readout against the repaired memory using the cached key, decodes a mask, and recomputes that frame's value. This value encoding is the only image-dependent cost; it is lightweight compared with the key encoder, and we will measure it. Finally, it evicts window entries that were not rebuilt and whose masks disagree with their rebuilt neighbours.

We choose $S$ by spending a budget of $k$ frames on the highest-entropy window entries. This option costs $O(k)$ decoder passes instead of the $O(d)$ full forward passes that replay would need. The budget $k$ and the selection rule are the parameters we ablate.

The request policy is a simple threshold rule. A request is sent when $\hat{H}_{t+\hat d}$, a linear extrapolation of the recent entropy trend over the expected delay $\hat d$, crosses a threshold $\tau$. We tune $\tau$ on LVOS training videos to match a fixed request budget. We deliberately avoid a learned policy at first, so that gains can be attributed to the rectifier.

Baselines

The naive reference is open-loop XMem with no corrections. It shows how much drift there is to fix.

The strongest existing method is Lazy-XMem's entropy-triggered SAM-HQ pseudo-correction, run in the same delayed channel. When a correction arrives, it inserts the mask at the current frame, as Lazy-XMem would if it ignored the delay. Lazy-XMem reports $\mathcal{J}\&\mathcal{F}$ of 64.3 on LVOS, and a 13-point [email protected] gain over XMem, with human interaction on 1.05% of frames (Strike the Balance). That figure includes human clicks, and our setting uses automatic correction only. We will therefore first reproduce its automatic-only variant with zero delay, and treat that reproduced number as the zero-delay target.

A second strong competitor is CL-MC-style hard reset on arrival: the window's memory is flushed and replaced with the stale corrected entry. CL-MC reports 76.52% AUC and 6.85% missing rate on its Harborview data (CL-MC). That dataset is not part of our protocol, so we reimplement the reset logic on LVOS.

We also run three further reference points. Fixed-interval requests mirror LTED-Ada's LTED-IntV baseline. Full replay re-runs XMem over every window frame and serves as the accuracy upper bound at maximum cost. Zero-delay correction is the ceiling.

Every method uses the same frozen XMem, the same SAM-HQ corrector, the same latency traces, and the same request budget, set so that requests go out on about 1–2% of frames. Each method's threshold gets an identical tuning budget on LVOS training videos.

We count the proposal as beating the field under two conditions. First, at the 1.45 s profile it must recover at least half of the $\mathcal{J}\&\mathcal{F}$ gap between hard reset and zero-delay correction. Second, it must stay within one point of full replay while using under 20% of replay's added compute. The decisive ablation keeps everything fixed but recomputes keys as well as values. If recomputing keys does not help, that confirms the claim that drift lives in the values.

Why not just…?

Why not just reset the tracker when the correction arrives? The corrected mask describes frame $t_s$, not the current frame. Inserting it as if it were current mismatches a d-frame-old object with the present image. Flushing the window removes the only record of how the object moved. This failure is unmeasured in the file, since CL-MC runs with no delay, and the hard-reset baseline at growing delay is precisely the experiment that measures it. If hard reset stays within a point of our method at all delays, the rectifier is unnecessary and we will say so.

Why not just request corrections on a fixed timer? LTED-Ada reports that its fixed-interval baseline is worse than adaptive offloading in delay and accuracy, because a timer ignores both scene motion and queue state. We include it anyway, because with cheap rectification a dense timer might become competitive. The equal-budget comparison settles this.

Why not just replay the window? Replay is the honest accuracy upper bound. It costs $d$ full forward passes per correction, though, which is tens of frames at edge latency, and those passes compete with live tracking on the same weak device. The comparison of compute against $\mathcal{J}\&\mathcal{F}$ is designed to show whether partial value rebuilding gives up meaningful accuracy.

Why not just run a better tracker locally and skip remote correction? CL-MC finds that standalone SAM2 and SAMURAI drift on hard video. LVOS finds that even strong trackers lose about 25% on long sequences, and that clean memory recovers up to 20%. Local capacity alone does not fix the self-poisoning loop. We still report a stronger local tracker as a reference, to show where the remaining headroom lies.

Proposed Experiments

The main benchmark is LVOS, for its minute-long videos and its $\mathcal{J}\&\mathcal{F}$ and [email protected] metrics, with DAVIS 2017 as a short-video sanity check. On DAVIS 2017, STM reports $\mathcal{J}$ Mean of 79.2 (STM), and delayed correction should neither help nor hurt much there.

The central figure plots $\mathcal{J}\&\mathcal{F}$ against median delay, from 0 to 5 s plus the heavy-tailed profile, for every baseline at matched request budget. Our prediction is specific. Hard reset should degrade roughly linearly in delay, because the stale mask grows more misaligned with every frame. The rectifier should stay close to replay until the delay window exceeds the tracker's memory horizon.

A second experiment splits the result by the frames where corrections arrive. It separates corrections that land after the object has moved substantially from those that land while it is stationary, since rectification should matter only for the former.

A third experiment compares the trend-extrapolated request rule with plain entropy thresholding at each delay. Extrapolation should help only at long delays; if it helps everywhere, it is just a better threshold.

Four results would falsify the claims. The first is key-plus-value recomputation clearly beating value-only. The second is hard reset matching the rectifier at 1.45 s. The third is uncertainty-selected rebuilding performing no better than rebuilding a uniformly subsampled set of $k$ frames. The fourth is the heavy-tailed profile collapsing every method to the open-loop baseline, which would mean that very long delays, of the kind seen in quantum cloud queues, are beyond what any correction scheme can handle. Any of these would be a useful finding about where late corrections stop being worth requesting.