A video object segmenter works one frame at a time. It takes the mask it predicted for earlier frames, stores it in a memory, and uses that memory to segment the next frame. This sets up a feedback loop. If the mask at frame $t$ bleeds a few pixels into the background, that bleed becomes part of the guidance for frame $t+1$. The model has no way to tell it apart from the real object, so the error can grow into "mask drift" over long videos. Boundary-weighted losses were built for a separate, static problem. These are training penalties that concentrate on the thin band of pixels where an object meets its surroundings, instead of averaging over mostly-interior pixels. So far they have been used almost entirely as per-frame sharpening tools.
This note proposes connecting the two in a specific way. During training, we deliberately corrupt the mask the model receives from the previous frame. We then apply a boundary-distance loss only in the band where that inherited error lives. This rewards the model for pulling a wrong upstream contour back toward the true edge rather than reproducing it. The claim is that drift is governed by a measurable per-frame "error gain" at object boundaries, meaning how much of an inherited boundary error survives into the next frame. We claim a loss aimed at that band can push the gain below one without changing the architecture or adding inference cost.
Core Claim and What Is New
Previous work touches each half of this, but not the combination.
Active Boundary Loss pushes predicted boundary pixels toward ground-truth edges using a distance transform. Fine-tuning a space-time memory network with it on DAVIS-2016 improved contour accuracy by 1.0%. However, it scores each frame independently. It never looks at what the model was handed from the previous frame, and it was not tested on long or multi-object videos.
Boundary loss for highly unbalanced segmentation provides the distance-map integral we reuse. It also supplies a constraint: that loss cannot train from scratch and must be ramped in alongside a region loss.
On the propagation side, Delving into the Cyclic Mechanism fights drift with an unweighted cyclic reconstruction loss plus test-time gradient correction, at roughly 9.3 FPS. Reliable Propagation-Correction Modulation fights drift by filtering the memory with Shannon entropy, and is trained with ordinary cross-entropy and Dice. Boundary-Aware Feature Propagation uses boundaries to stop error spreading, but spatially within one image and through an architectural gate rather than a loss.
What none of these do is define the loss weight relative to the upstream error. That is, none ask "where was the mask we were given wrong?" and supervise geometry precisely there. That conditioning is the new ingredient. The note stands or falls on showing that it matters beyond two simpler things: training on corrupted inputs with a plain loss, and adding a boundary loss everywhere.
The Loss
Let $\hat M_{t-1}$ be the mask propagated into frame $t$. It is either the model's own prediction or a synthetically perturbed ground truth. Let $G_t$ be the ground-truth mask at frame $t$, and $\phi_{G_t}$ its signed distance map (negative inside, positive outside). Define the inherited-error band $B_t$ as the symmetric difference between $\hat M_{t-1}$ warped to frame $t$ and $G_t$, dilated by a few pixels. With $s_\theta(x)$ the predicted foreground probability at pixel $x$, the training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{region}} + \lambda(\tau)\,\frac{1}{|B_t|}\sum_{x \in B_t} \phi_{G_t}(x)\, s_\theta(x), $$
Here $\mathcal{L}_{\text{region}}$ is the base model's usual cross-entropy and Dice loss. $\lambda(\tau)$ is a weight that ramps up from zero over training step $\tau$, as Kervadec et al. require for stability. The second term is their boundary integral restricted to the inherited-error band. It charges the model for keeping foreground probability on pixels that the upstream mask wrongly included, in proportion to how far outside the true contour those pixels lie. Symmetrically, it charges for missing pixels that were wrongly excluded.
Two known tensions shape the design. First, the DAVIS-2017 challenge report shows that the dominant multi-object failure is identity switching. Foreground-versus-background scores exceed 82% while per-object scores sit near 70%. A purely geometric pull could therefore snap a contour onto the wrong neighbouring dancer. To guard against this, $\phi$ is computed per object, and pixels belonging to another instance are treated as "outside" at full distance, so the term penalizes, rather than rewards, leakage into neighbours. Second, when an object is fully occluded, $G_t$ has no boundary. In that case the band term is dropped for that object and only the region loss acts, which avoids the gradient blow-ups noted in the file.
Research Program
The work has three parts.
The first is a diagnostic that the field lacks: a controlled measurement of boundary error gain. On DAVIS-2017 validation, we take a trained segmenter and inject calibrated perturbations into its memory mask at a chosen frame. The perturbations are dilation, erosion, rigid shift, and a spurious blob attached to the contour, at 2–16 pixels. We then track the boundary-band error over the next 30 frames. The ratio of error at $t+k$ to error at $t$ gives an empirical gain curve. A curve that rises means the model amplifies inherited errors; one that falls means it corrects them. We measure this for XMem and AOT first. Whether today's strongest models amplify or already contract small boundary errors is itself an informative result. If they already contract strongly, the proposal's premise is weak and we will say so.
The second part is training. During the multi-frame clips that memory-based segmenters already train on, a fraction of the propagated masks are replaced by perturbed versions drawn from the same distribution as the diagnostic. The band loss above is then applied. Distance maps for $G_t$ are computed on the GPU per batch, which avoids the storage cost noted for precomputed maps.
The third part is analysis. We relate the change in gain to the change in long-video accuracy and to identity-switch counts, to test whether lowering boundary gain actually explains reduced drift.
Baselines
The naive reference point is the unmodified base model, XMem XMem, retrained by us with its published recipe on our hardware. The file reports 86.2 $\mathcal{J}\&\mathcal{F}$ on DAVIS-2017, 85.7 overall score on YouTube-VOS 2018, and 90.0 on the Long-time Video dataset. It is the strongest competitor in the file and already built specifically to resist long-horizon decay. Our reproduction, not the paper number, is the comparison point, and all variants share its data, schedule, and seeds. We run three seeds, because gains near one point are within noise for this benchmark. AOT (R50-AOT-L, 84.9 on DAVIS-2017 per the file) serves as a second backbone, to show the effect does not depend on one memory design.
The main competing idea is per-frame boundary supervision. We fine-tune the same base model with Active Boundary Loss over whole frames, as Wang et al. did for STM. We give it an identical ramp schedule and a tuning budget matched to ours, at five settings for its weight.
The strongest drift-specific methods are cyclic training with gradient correction (Li et al., 72.3 $\mathcal{J}\&\mathcal{F}$ on DAVIS-2017 at about 9.3 FPS) and P2C (Xu et al., 79.2 on DAVIS-2017 test-dev and 83.9 on YouTube-VOS 2019). These are older backbones, so we compare to them on gain curves and relative improvement over their own base, not on raw score. We also reimplement the cyclic loss on our base model so that it shares our training budget.
Two ablations isolate the key ingredient. The first uses perturbed-input training with the plain region loss and no band term. The second applies the band term on a randomly placed band of equal area rather than on the inherited-error band. Beating the base model means more than one point in the contour score $\mathcal{F}$ on DAVIS-2017, with the gain growing on the Long-time Video dataset, and a measured gain below the base model at every perturbation size, all at unchanged inference speed. Beating the ablations by a margin larger than the seed spread is what would justify the error-conditioned weighting specifically.
Why not just…?
Why not just train on the model's own corrupted predictions with the usual loss? This is the most serious threat. Exposure to one's own errors alone might teach correction, with no boundary geometry involved. It is exactly the first ablation. If that ablation matches the full method on gain curves and long-video accuracy, the conclusion is that the band loss adds nothing. The note would then report a perturbation-training result instead.
Why not just add Active Boundary Loss to every frame? Its reported benefit in video is a 1.0% contour gain on single-object DAVIS-2016 (Wang et al.). It spends its gradient on every contour, including ones the model already gets right, and it has no notion of inherited error. Our gain-curve diagnostic tests directly whether per-frame sharpening changes how errors propagate or only how the first frame looks.
Why not just correct masks at test time or filter the memory? Gradient correction works but runs at about 9.3 FPS (Li et al.), against 22–30 FPS for XMem and 51.4 FPS for AOT-T. The entropy filter in P2C judges pixel label uncertainty, not contour position, so a confidently wrong edge passes it. Our method costs nothing at inference. It is also complementary, so we additionally report it combined with the entropy filter.
Why not rely on bigger memory, as XMem already does? XMem's multi-store memory reduces forgetting. However, its working memory still stores the model's own masks, so a wrong contour enters memory as easily as a right one. The perturbation diagnostic on XMem shows whether memory design alone already yields contraction. If it does, our training should show no added benefit there, and the result will be reported as such.
Decisive Checks
The idea is supported only if three things hold together.
First, the full method lowers the empirical boundary error gain relative to both ablations and to per-frame Active Boundary Loss, across perturbation sizes and on both backbones.
Second, the improvement over reproduced XMem is larger on the Long-time Video dataset than on DAVIS-2017. That is the signature of reduced compounding rather than better single-frame edges.
Third, identity-switch counts on DAVIS-2017 multi-object sequences do not increase, which confirms that per-instance distance maps avoid snapping onto neighbours. We also report YouTube-VOS 2018 seen and unseen categories, to check that learned correction is not category-specific.
If the gain drops but long-video accuracy does not move, we will conclude that boundary drift is not what limits current memory models and report that negative result. The same diagnostic-plus-band-loss recipe transfers naturally to multi-turn image editing on MagicBrush, where FreqEdit documents edge collapse after about five turns. That extension is deliberately left for after the video result is settled.
References
- Active Boundary Loss for Semantic Segmentation
- Boundary loss for highly unbalanced segmentation
- Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation
- Reliable Propagation-Correction Modulation for Video Object Segmentation
- Boundary-Aware Feature Propagation for Scene Segmentation
- The 2017 DAVIS Challenge on Video Object Segmentation
- XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
- Associating Objects with Transformers for Video Object Segmentation
- MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing
- FreqEdit: Preserving High-Frequency Features for Robust Multi-Turn Image Editing