When a user segments an object in a video with a promptable tracker such as SAM 2 (the Segment Anything Model 2, which turns a click or box into a mask and carries that mask forward frame by frame), the first prompt does two jobs. It defines the object, and it becomes a stored memory that every later frame consults. Single-image studies show the first job is fragile. Moving a click by one pixel can drop IoU (the overlap between predicted and true mask) from 95% to 60%, and real users click about 15% of the object diagonal away from where benchmark scripts click (TETRIS). Video studies show the second job is dangerous, because corrupted masks written into memory keep degrading later frames (SAM2Long). Nobody has measured what happens when these two effects combine: a slightly wrong first click produces a slightly wrong first mask, and that mask is stored as a permanent reference.
We propose to measure this and to fix it. The measurement is a persistence curve: how much tracking quality is lost at each frame index because of a small, realistic perturbation of the first-frame prompt. The fix follows from a specific hypothesis. Later frames often contain evidence that would correct the initial mistake, for example the object moving away from a neighbour it was fused with, or a boundary halo disappearing under a new viewpoint. The prompted frame's memory is never evicted, though, so it keeps pulling predictions back toward the original error. The method, revisable anchoring, runs a short backward propagation from a later frame to the prompted frame without the anchor memory. Where the backward mask and the original mask disagree, and the disagreement holds up under a stability test, the anchor memory is rewritten. The whole process is training-free.
Core Claim
Our first claim is that, for SAM 2 under human-like first-frame prompt noise, the per-frame quality gap between clean and perturbed prompts does not decay toward zero. It partially recovers and then plateaus, and the height of that plateau tracks how much attention later frames pay to the prompted frame's memory. Our second claim is that rewriting the anchor using backward-propagated temporal evidence lowers the plateau more than any fix that uses only frame 0 and more than forward multi-hypothesis tracking at matched compute. We expect this for errors that are local: boundary leakage, halo, or merging with an adjacent object. We do not expect it for semantic errors such as choosing a part instead of the whole, where the backward pass will simply inherit the wrong object. We state that limit in advance and test it separately.
We define the persistence curve as
$$\Delta(t) = \mathrm{JF}_{\text{clean}}(t) - \mathbb{E}_{p \sim \mathcal{P}}\left[\mathrm{JF}_{p}(t)\right],$$
where $\mathrm{JF}(t)$ is the standard region-and-boundary score J&F at frame $t$, "clean" is the benchmark's scripted first-frame prompt, and $\mathcal{P}$ is a distribution of perturbed prompts matched to the TETRIS human click-offset statistics. The quantity we care about is the late-frame plateau $\Delta_\infty$, the mean of $\Delta(t)$ over the final third of each sequence.
How this differs from prior work
The prompt-sensitivity literature fixes the first mask in isolation. Stable-SAM adds small deformable-attention modules that pull decoder attention back onto the target under sloppy boxes. SAMRefiner extracts noise-tolerant prompts from coarse masks for one-shot cleanup. A-SAM turns prompt sensitivity into a distribution over plausible masks. All three work on a single image. None of them can use evidence that only appears in later frames, and that later evidence is the resource our method relies on.
The sequential side works in the other direction. SAM2Long keeps several forward pathways and prunes them by accumulated predicted IoU. That protects the recent-frame memory, but every pathway shares the same prompted-frame anchor, so an anchor error is common to all branches and branching cannot remove it. VRR shows in video generation that compounding error has an internal signature that appears before visible failure. We borrow that diagnostic idea in the form of attention mass on the anchor, but we do not regularize representations.
Interactive methods that feed masks back across clicks show the same structural risk inside a single image. RITM feeds the previous mask back as input. EMC-Click freezes first-click features for all later corrections. SPT retrieves past clicks and masks across image sequences. None of these three evaluates what happens when the earliest prompt is noisy. The restoration results (joint demosaicing and super-resolution, GFN) supply the general lesson that downstream stages treat upstream artifacts as real signal. They also constrain our design: their answer is joint end-to-end retraining, which we avoid, because Stable-SAM's comparison with full tuning suggests that retraining the foundation model is costly and risks forgetting.
Research Program
The work has three stages.
The first stage is diagnosis. We run SAM 2 on DAVIS-2016, SA-V, and ADE20K-Seq. The first-frame prompts are drawn from three sources: the TETRIS-calibrated human offset distribution, noisy boxes in the style of the Stable-SAM protocol, and worst-case clicks found by TETRIS-style white-box search on frame 0. For each run we record $\Delta(t)$, the plateau $\Delta_\infty$, and the fraction of memory-attention mass that later frames place on the prompted frame. We then label each perturbed first mask as a local error or a semantic error. A mask counts as local if its IoU with the ground truth exceeds 0.5 and the error region touches the true boundary. Otherwise it counts as semantic. The prediction is that local errors show partial recovery followed by a plateau, and that frames placing more attention on the anchor retain more of the frame-0 error.
The second stage is the method. After forward tracking reaches frame $K$, with $K$ tuned in the range 10 to 30, we seed a backward pass from frame $K$ to frame 0. This pass uses only non-anchor memories. It produces a candidate mask $\hat m_0$ for frame 0, which we compare with the original anchor mask $m_0$. The anchor is rewritten with $\hat m_0$ only if $\hat m_0$ is stable, meaning that backward passes seeded from two different frames $K$ and $K'$ agree above a threshold. The agreement test guards against replacing one noisy mask with another. The cost is a fixed one-time overhead of about $2K$ frame passes. We also test a periodic variant that re-checks the anchor every $K$ frames.
The third stage is analysis. We check whether the rewrite gate fires mostly on local errors, and whether it ever damages clean-prompt runs. The clean-prompt $\mathrm{JF}$ must not drop.
Baselines
The naive reference is SAM 2 with the perturbed prompt and no correction. Its $\Delta_\infty$ defines the problem size. There is no reported result on Lacuna for SAM 2 under perturbed first-frame prompts, so this number is itself a contribution. SAM 2 with the clean prompt gives the ceiling.
The strongest existing sequential method is SAM2Long. It reports an average J&F gain of 3.7 points over SAM 2 across 12 video benchmarks and 5.3 points on SA-V, with about a 14% FPS slowdown for 3 pathways (SAM2Long). Those numbers are for clean prompts. We will rerun it under our perturbations and give it the number of pathways that matches our wall-clock budget. Beating it means achieving a lower $\Delta_\infty$ at equal or lower compute. We will also test the two methods combined, since they target different memories and could plausibly stack.
The strongest frame-0 method is Stable-SAM-style correction. The reported comparison is noisy-box mIoU rising from 48.8 for SAM to 82.3 for Stable-SAM, in a single-image setting (Stable-SAM). We apply its plugins to SAM 2's image decoder on frame 0 only, trained for the one HQSeg-44K epoch the paper uses, and track from the corrected mask.
The ablations isolate the key ingredient, which is revising the anchor using temporal evidence. In the first ablation, the backward mask is added as an extra memory rather than replacing the anchor. In the second, the anchor is always rewritten, with no stability gate. In the third, the rewrite uses a frame-0 jitter consensus in place of the backward mask. If the third ablation matches the full method, temporal evidence is not what is doing the work.
Why not just…?
Why not just jitter the first prompt, run several passes, and keep the most stable mask? This is the most serious threat. The usual objection to test-time ensembling is per-step latency, which EMC-Click emphasizes, but that objection does not apply here because the ensemble runs on one frame only. The objection from A-SAM and TETRIS, that averaging blurs boundaries across a spiky sensitivity landscape, applies to averaging but not to selecting the modal mask. We therefore run the strong version: select the mask most stable under jitter, at the same number of passes as our backward window. If it matches revisable anchoring on local errors, our temporal-evidence hypothesis is wrong for those errors and we will report that. Our expectation is that it helps on halo errors but not on merges with adjacent objects, because on a single frame the merged region is equally stable under jitter.
Why not just fine-tune SAM 2 on noisy first-frame prompts? VRR reports that more sequential training without regularization can speed up representational shortcuts, and Stable-SAM reports that full tuning costs more epochs and risks zero-shot ability. Both results are from neighbouring settings, not this one. We treat fine-tuning as a secondary comparison at equal GPU-hours on SA-V, and we check held-out benchmarks for forgetting.
Why not just trust the predicted IoU and drop low-confidence frames? SAMRefiner found that SAM's IoU head is unreliable on degraded inputs, and SAM2Long found that greedy confidence filtering fails under occlusion. More fundamentally, confidence filtering governs recent-frame memory and never touches the anchor. We include it as a baseline, and we expect it to leave $\Delta_\infty$ unchanged.
Proposed Experiments
The decisive checks are three. First, the diagnostic must show a nonzero plateau $\Delta_\infty$ that correlates with anchor attention. If SAM 2 already self-heals, with $\Delta_\infty \approx 0$, the proposal stops at the measurement paper. Second, on local-error runs, revisable anchoring must reduce $\Delta_\infty$ more than both jitter-selection and SAM2Long at matched compute, and more than the add-as-memory ablation. Third, on semantic-error runs, we predict little gain, and on clean prompts we require no loss.
All three checks use DAVIS-2016, SA-V, and ADE20K-Seq with TETRIS-calibrated and adversarial prompts. We report J&F per frame bin, $\Delta_\infty$, the rate at which the gate fires, and FPS. We also run a small transfer study on multi-turn click sessions in the TETRIS benchmark. There, a mask obtained later in the session, with the first click removed, plays the role of the backward pass. This tests whether the same anchor-persistence effect appears in RITM-style mask feedback.
References
- TETRIS: Towards Exploring the Robustness of Interactive Segmentation
- SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree
- Stable Segment Anything Model
- SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement
- Flaws can be Applause: Unleashing Potential of Segmenting Ambiguous Objects in SAM
- Mitigating Compounding Error via Video Representation Regularization
- Reviving Iterative Training with Mask Guidance for Interactive Segmentation
- Efficient Mask Correction for Click-Based Interactive Image Segmentation
- SPT: Sequence Prompt Transformer for Interactive Image Segmentation
- Deep Residual Network for Joint Demosaicing and Super-Resolution
- Gated Fusion Network for Joint Image Deblurring and Super-Resolution