Delayed-Commit Memory: Future-Verified Identity Memory with Zero Output Latency

Video object segmentation (VOS) trackers follow a few objects through a video: given each object's mask in the first frame, they must segment it in every later frame. Bidirectional video models use future frames to settle ambiguous moments, such as which of two similar objects came out from behind an occluder. The cost is latency. The system cannot produce an answer for frame $t$ until frame $t+k$ has arrived. Our proposal rests on one observation: output latency and memory latency are separable. A tracker has to emit its mask for the current frame immediately, but nothing forces it to write that frame into its long-term memory immediately. We propose to emit masks with zero look-ahead and to delay only the memory write by $k$ frames. During those $k$ frames, the model checks the pending entry against the future frames it has since seen, relabels it if two identities were swapped, and discards it if it was wrong.

This matters because the evidence in the literature says memory quality, not per-frame segmentation, is what caps long-video tracking. On the LVOS long-video benchmark, the DDMemory tracker reaches 61.9 $\mathcal{J}\&\mathcal{F}$, and simply giving it ground-truth-quality memory masks raises performance by up to 20.8 points (LVOS: A Benchmark for Long-term Video Object Segmentation). Here $\mathcal{J}$ is mask overlap and $\mathcal{F}$ is boundary accuracy. Bidirectional verification is the classic way to obtain cleaner labels. If we spend it on the memory rather than on the output, we can attack that oracle gap without paying the frame-buffering delay that rules bidirectional methods out of closed-loop use.

Core Claim

A tracker whose memory entries are committed $k$ frames late, after a forward–backward consistency check, will recover a substantial fraction of the LVOS oracle gap. It will also cut identity swaps on MOSE, the benchmark built from crowded scenes with frequent disappearances. Its emitted masks keep strictly zero look-ahead, and its memory cost stays bounded. The claim goes beyond "gate memory by confidence." We claim that future-verified writes beat causal confidence-gated writes that use the same memory budget and compute, and that the advantage concentrates on reappearance and crossing events.

Relation to Prior Work

Three lines of work motivate the proposal by showing what future context buys, and all three spend it on the output. BOFP warps features forward and backward from sparse keyframes. It reaches 86.5% mean temporal consistency on Cityscapes, and its backward branch alone adds nearly 4% accuracy, but it must wait for the next keyframe (Temporally-Consistent Video Semantic Segmentation with Bidirectional Occlusion-guided Feature Propagation). CoVOS propagates masks across bidirectionally coded frames using compressed-video motion vectors. Its authors state that it inherits the codec's look-ahead and suits offline use (Accelerating Video Object Segmentation with Compressed Video). Tube-Link links 2–6-frame subclips with cross-tube attention, which buffers up to about 200 ms at 30 FPS and links only adjacent tubes (Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation). We keep their core insight, that the future disambiguates the past, but no emitted mask ever waits on the future.

Two further lines constrain the design. Causal memory trackers show where errors come from. DDMemory compresses history into a recurrent state with a constant 0.88 GB footprint, and that compression is exactly where drift enters. The streaming mixture-model approach, which keeps time-decaying cluster statistics of each object's features, lets an occluded object's statistics erode, and its authors report failure when similar objects cross (From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models). DeAOT separates appearance features from per-object identity features, yet it still swaps identities under mutual occlusion and falls from 85.2% on DAVIS 2017 to 59.4% on MOSE (Decoupling Features in Hierarchical Propagation for Video Object Segmentation; MOSE: A New Dataset for Video Object Segmentation in Complex Scenes). These results tell us that fixed compression and decay are not enough. They also tell us the fix must target what gets written, rather than adding capacity.

The closest causal competitors are memory-curation schemes on top of SAM2, the promptable segmentation foundation model with a streaming memory. SAMITE calibrates which frames enter memory (SAMITE: Position Prompted SAM2 with Calibrated Memory for Visual Object Tracking), and a training-free surgical variant augments SAM2's memory selection (Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation). Both decide at write time using only the past. Our method decides after seeing $k$ more frames. The key ablation below tests whether that difference matters.

The latency framing comes from streaming speech. Wait-$k$ policies delay output by a fixed number of tokens to gain context (Training Simultaneous Speech Translation with Robust and Random Wait-k-Tokens Strategy). We apply wait-$k$ to the model's memory rather than to its output.

Research Program

The module wraps an existing causal tracker. We will use DeAOT, because its appearance branch already computes frame-to-frame attention maps we can reuse cheaply, and a DDMemory-style tracker, because it defines the LVOS oracle gap. At each frame the tracker emits its mask as usual. Its features and predicted identity masks go into a pending queue of length $k$ rather than into long-term memory. When frame $t$ arrives, the entry for frame $t-k$ is re-examined. We propagate the current masks backward from $t$ to $t-k$ by reusing the stored attention maps in reverse. For each object $i$, we compute a commit score

$$s_i = \mathrm{IoU}\big(\hat m_i^{\rightarrow}(t-k),\ \hat m_i^{\leftarrow}(t-k)\big),$$

where $\hat m_i^{\rightarrow}$ is the mask the tracker originally emitted for object $i$ at frame $t-k$, and $\hat m_i^{\leftarrow}$ is the mask obtained by carrying frame $t$'s prediction back to $t-k$.

Three decisions follow from these scores. First, when two or more objects overlap within the pending window, we solve a small assignment over their identities that maximizes total forward–backward agreement. If the optimal assignment permutes labels, we relabel the pending entry before committing it. This is the swap repair that DeAOT lacks. Second, entries whose best score falls below a threshold are dropped rather than written. Third, if the look-ahead confirms that an object has been absent across the whole window, that object's long-term slots are frozen: no updates and no decay, so its identity descriptor survives the occlusion intact. This directly targets the erosion failure in recurrent and mixture-model memories.

Memory stays bounded. The pending queue holds $k$ frames of features, and long-term memory keeps whatever fixed budget the base tracker uses. Compute adds one backward attention pass per frame, which we will measure and report. For downstream uses such as object counting that can tolerate delay, the same machinery also produces a revised track at delay $k$, and we will report it separately from the zero-latency output.

The study sweeps $k \in \{0, 2, 4, 8, 16, 30\}$ frames. At $k=0$ the method reduces to the base tracker. The sweep traces how memory accuracy trades against memory latency while output latency stays pinned at zero. We will also directly measure write quality under each policy, as the IoU of committed memory masks with ground truth, which checks whether the mechanism works for the reason we claim.

Baselines

The do-nothing reference is the unmodified base tracker writing to memory immediately, which is the $k=0$ point. For LVOS the strongest reported number in our file is DDMemory at 61.9 $\mathcal{J}\&\mathcal{F}$ on the validation set (LVOS: A Benchmark for Long-term Video Object Segmentation). For MOSE it is DeAOT at 59.4% (MOSE: A New Dataset for Video Object Segmentation in Complex Scenes). We will rerun both under our own evaluation to control for implementation differences. Beating them means a gain on the rerun numbers with identical weights, identical memory budget, and output latency of zero.

The strongest causal competitor is confidence-gated writing: the same base tracker writes a frame to memory only if its own confidence, or a learned quality score in the spirit of SAMITE's calibrated memory, exceeds a threshold. The comparison is fair only if gating is given the same total memory budget, the same threshold-tuning budget on a held-out split, and a matched compute budget. We will therefore also give the gated version an extra refinement pass of equal cost. Delayed commit must beat it by a clear margin on reappearance and crossing frames. Otherwise the future-verification ingredient is not earning its keep.

The strongest bidirectional competitor is output-level look-ahead: the same tracker emits its mask for frame $t$ only after seeing $t+k$, in the Tube-Link or BOFP style. This is an upper-bound-style comparison at matched $k$. Our method pays no output latency, so matching most of this baseline's accuracy counts as success.

Two ablations isolate the key ingredient. The first removes swap repair and keeps only commit or drop. The second removes dormant freezing. A third ablation replaces the forward–backward score with a causal-only score computed on the same delayed schedule. This separates "writing late" from "writing after checking the future."

The oracle-memory result on LVOS defines the ceiling. We report the fraction of the gap closed as (ours − base)/(oracle − base), with the oracle rerun on our base tracker.

Why not just…?

Why not just gate memory writes by the tracker's own confidence? This is the real threat. Confidence gating costs nothing in latency and may catch many bad writes. Its blind spot is the failure we target. During an identity swap between similar objects, the tracker is often confidently wrong, and DeAOT's reported swap failures under mutual occlusion suggest exactly this. No past-only score can tell which object emerged along which path. The matched-budget comparison, broken down by event type, settles the question. If gating matches us on crossing and reappearance frames, we will conclude that the future-verification mechanism is unnecessary and report that.

Why not just buffer a few frames at the output, as Tube-Link does? Such a buffer delays every mask. That is ruled out in the closed-loop settings motivating this work, where papers on video segmentation acceleration describe look-ahead delay as the deployment blocker (Fast Semantic Segmentation on Video Using Block Motion-Based Feature Interpolation). Our experiments report both methods at equal $k$, so the reader sees exactly how much accuracy output buffering buys over memory-only buffering.

Why not distill a bidirectional teacher into a causal student? Distillation improves per-frame features. It cannot give the student information that does not yet exist at inference time: the student still has to guess which of two similar objects is which at the moment they separate. It also requires retraining for every base tracker, whereas our module is training-light and wraps existing trackers. We will not claim that distillation fails. We note only that it is complementary, and a distilled student could itself serve as the base tracker.

Why not just keep more memory, as uncompressed memory banks do? Unbounded banks grow linearly with video length and crash on long videos. Larger memories also do not prevent contaminated entries: XMem, which uses such a bank, still falls to 57.6% on MOSE. Memory size is held fixed across all our comparisons, so any gain comes from what is written rather than how much.

Proposed Experiments and Decisive Checks

The primary evaluations are LVOS validation, for long absences and reappearance, and MOSE validation, for crowding, crossings, and a 28.8% disappearance rate. DAVIS 2017 validation serves as a sanity check that short-video accuracy does not regress. There, DeAOT reports 85.2 and XMem 86.2 $\mathcal{J}\&\mathcal{F}$.

The metrics are $\mathcal{J}\&\mathcal{F}$ at zero output latency, $\mathcal{J}\&\mathcal{F}$ restricted to the 30 frames after each reappearance, identity-swap counts at crossing events, committed-memory IoU, FPS, and peak GPU memory. We will also report $\mathcal{J}\&\mathcal{F}$ of the delayed revised track at each $k$.

Three results would be decisive. First, the fraction of the LVOS oracle gap closed should rise with $k$ and saturate. If it stays flat, memory contamination is not the mechanism behind the gap. Second, delayed commit must beat matched-budget confidence gating specifically on crossing and reappearance frames on MOSE. Third, committed-memory IoU should rise alongside tracking accuracy. If accuracy improves without cleaner writes, our explanation is wrong even if the numbers look good.