Online video segmentation models find, outline and track every object in a video one frame at a time. Most recent ones do this by carrying an "object query" forward from frame to frame. An object query is a learned vector that summarizes one object; the model decodes it into a class label and a mask, then reuses it to find the same object in the next frame. That single vector is asked to do two things that conflict. It must keep a stable sense of what the object is, even as the object moves, turns or is partly hidden. It must also track where the object's boundary is, down to the pixel, and that changes every frame. In still images, splitting these two jobs into separate prediction paths is known to help. In video, the carried-forward vector still mixes them, and the mixing is repeated hundreds of times. A second problem compounds the first: the visual detail an object needs depends on its size. A surgical thread a few pixels wide needs high-resolution features, while an organ filling the frame needs coarse context, and the same object can move between these regimes within a single clip.
We propose to split each propagated query into two parts. The first is a slowly changing identity part that never receives gradients from the boundary losses. The second is a fast spatial part that predicts the object's box. That predicted box then decides where, and at which resolution, the spatial part looks in the next frame. The model opens a small local attention window around the predicted location, on whichever level of the multi-resolution feature map makes the object span a roughly fixed number of feature cells. The result is that thin and tiny objects are processed on high-resolution features at a fixed cost per object, large objects get enough context, and identity is shielded from frame-to-frame boundary noise. Two claims are tested separately: that the split reduces long-horizon identity drift, and that size-matched local windows improve masks for thin and small structures more cheaply than global or sparse attention.
Core Claim
We make two claims, and they can fail independently. The first is that stopping gradients from the localization losses into a dedicated identity sub-query reduces identity switches and drift on long, heavily occluded videos, beyond what separate output heads on a shared query achieve. The second is that dense self-attention inside a per-object window works better and costs less than global attention, sparse deformable sampling, or a fixed backbone window, for thin and small objects. The window must be placed by the propagated box and sized by choosing the pyramid level, not by changing the window's token count. Concretely, for an object whose predicted box at frame $t$ has width $w$ and height $h$ in pixels, we read features at the level
$$ l^{*} = \arg\min_{l} \left| \log_2 \frac{\sqrt{wh}}{s_l} - \log_2 m \right|, $$
where $s_l$ is the downsampling factor of pyramid level $l$ (4 to 32) and $m$ is the target object width in feature cells (for example 8). We then attend densely within an $M \times M$ window ($M \approx 2m$, to include surrounding context) centered on the box. Because the window always holds $M^2$ tokens, the cost per object is constant whatever the image resolution or object size. Total cost grows with the number of tracked objects rather than with the square of the pixel count.
Relation to Prior Work
Several papers motivate the split. In still-image detection, separating classification from localization branches resolves conflicting gradients, as shown by Lu et al. and YOLO-ERF. DeCo-DETR goes further, isolating localization gradients from category alignment with stop-gradient operators in an open-vocabulary detector. Decoupled PROB separates objectness from classification across decoder layers. All of these are single-frame, so none tests whether the benefit compounds when the representation is carried forward in time. That is the question here.
Online propagation methods supply the base we build on. InsPro propagates {query, box} pairs with a deduplication loss for new objects. InstanceFormer propagates queries, reference points and class scores with a memory queue. DVIS separates segmentation, tracking and refinement into modules but tracks on compact query vectors only. InsPro's propagated box is already a crude spatial stream, so our novelty is not "propagate a box". It has two parts. The identity and spatial vectors are gradient-isolated sub-queries, which InsPro and InstanceFormer do not have: they update class and position jointly in one vector. And the propagated box is used to place and scale a dense attention window, which none of these papers do.
QueST documents "silent semantic drift" in long-horizon propagation and counters it with persistent semantic anchors plus kinematic constraints on point tracking. Our identity sub-query plays a similar anchoring role, but it comes from the training signal rather than from added constraints.
VidEoMT is the counterpoint. It deliberately couples tracking and segmentation inside one large pre-trained ViT and reports 68.6 AP on YouTube-VIS 2019 at 160 FPS. This suggests that, at large scale, the coupling we target may not matter. We address this head-on with a scale-controlled comparison below.
The multi-scale side is shaped by the surgical-video line. TAFE separates organ and tool geometry with square versus strip convolutions on still frames. TAFPNet adds temporal query propagation over an asymmetric pyramid, but with global attention and no task decoupling. TAFPNet is therefore the closest existing system: it shares two of our four directions and our main application domain, and it differs exactly in the window mechanism and the gradient split. Work on scale failure explains why fixed-resolution processing struggles. Ndipenoch et al. show downsampling erases small structures, Nichyporuk et al. show large structures dominate gradients, and Guo shows small receptive fields lose large-structure context.
Window attention itself comes from the shifted-window line surveyed in A Survey on Vision Transformer, and from the evidence in window-based attention for compression that dense local neighborhoods carry the boundary detail. Those windows sit on fixed grids. Ours are placed and scaled by the tracked object.
Research Program
We build on an InsPro-style online tracker with a ResNet-50 or Swin-T backbone and a four-level feature pyramid (1/4 to 1/32). Each tracked object carries two vectors.
The identity sub-query is updated once per frame by cross-attention to coarse (1/16 and 1/32) features inside an enlarged region around the object, plus attention over a short memory of its own past states. The classification loss and an identity contrastive loss train it. Stop-gradients block every box, mask and boundary loss from reaching it.
The spatial sub-query reads the identity sub-query through a stop-gradient. It predicts the box and then runs two or three layers of dense self- and cross-attention inside the size-matched window. The window is shifted by half its width on alternating layers, so tokens at its edges also interact. The spatial sub-query finally produces the mask through a dot product with high-resolution window features, and the mask is pasted back into the full frame. When the box crosses a pyramid level between frames, the window simply switches levels, so an object's scale change becomes a level change rather than a mismatch.
New objects are found by a separate set of learnable queries on the coarse levels, trained with InsPro's deduplication loss. Each newly found object is promoted into the decoupled pair once matched. This keeps discovery separate from propagation, in the spirit of Decoupled PROB.
We also study analytically and empirically how the two gradient signals interact. We measure the cosine between the classification and localization gradients on a shared query during training, as a function of sequence length and object scale. If the conflict is small, the first claim loses its motivation, and we would report that.
Baselines
The naive reference point is per-frame segmentation followed by association. Concretely, this means the same backbone and a Mask2Former-style segmenter run independently per frame, linked by Hungarian matching on mask IoU and embedding similarity, as in Tracking Without Bells and Whistles. It shows how much any propagation buys at all.
The strongest directly comparable online propagation baselines are InsPro and InstanceFormer. InsPro reports 43.2 AP on YouTube-VIS 2019 with ResNet-50 and COCO pre-training at 26.3 FPS, and 37.6 AP on YouTube-VIS 2021. InstanceFormer reports 40.8 AP on YouTube-VIS 2021 and 20.0 AP on OVIS. DVIS reports a 7.3 AP gain over the previous best on OVIS, but the file gives no absolute number for it (no reported result found on Lacuna for DVIS OVIS absolute AP). On surgical video the strongest reference is TAFE at 77.5% mIoU on EndoVis 2018, with TAFPNet reporting a further relative gain (+16.4% mIoU) but no absolute figure in the file.
We will retrain InsPro, InstanceFormer and DVIS from their released configurations with our backbone, input resolution, pre-training and schedule. Each method gets the same hyperparameter search budget, roughly a dozen runs. Published numbers are reproduced first, to confirm the retraining is faithful. Beating these baselines means a gain of at least 1.5 AP on OVIS and on its heavily occluded subset ($AP_{HO}$), a reduction in identity switches, and equal or better FPS, all averaged over three seeds with non-overlapping intervals. On EndoVis 2018, it means at least 2 mIoU over our retrained TAFE/TAFPNet-style global-attention baseline, concentrated in the thread, needle and clasper classes.
Three ablations isolate the ingredients. "Stop-gradient only" applies DeCo-DETR's gradient isolation to InsPro's single query, with no split and no windows. "Split, no window" keeps both sub-queries but uses InsPro's deformable attention. "Window, no split" keeps size-matched windows but a single coupled query. A fourth ablation fixes the window to one pyramid level. This separates "local dense attention" from "scale-selected level".
Why not just…?
Why not just add a stop-gradient between the class and box heads of an existing tracker? This is the most serious threat to the first claim. If DeCo-DETR-style isolation on InsPro's shared query gives the same drop in identity switches, then the split sub-queries are unnecessary. Our "stop-gradient only" ablation measures exactly this. The reason to expect a difference is mechanical. With one shared query, the box loss still rewrites the vector that is carried forward, even if it cannot reach the class head's parameters. Our split keeps localization updates out of the identity state entirely. If the ablation matches the full split, we will report that the cheaper fix suffices and narrow the claim to the window component.
Why not just use deformable attention with the propagated reference point, as InstanceFormer does? Deformable attention already offers linear cost and multi-scale sampling. However, it reads a handful of points per level, and the file reports that deformable baselines collapse on OVIS's heavily occluded subset (as low as 5.6 $AP_{HO}$ per the OVIS paper). For a thread a few pixels wide, a few sampled points can easily miss the structure. The "split, no window" ablation compares the two directly at matched FLOPs, using boundary metrics on thin classes rather than overall AP alone, since the advantage should appear there if anywhere.
Why not just use RoIAlign crops with a mask head, as in classic two-stage segmenters? Cropping and resizing a box region also gives a fixed-size, scale-normalized input, and it is a real competitor to the window idea. The differences are that resizing interpolates thin structures away, whereas our level selection reads native-resolution cells, and that our window keeps surrounding context. We will replace the window attention with RoIAlign at the same token budget plus an equal-depth conv mask head. If it matches us, the contribution is scale-normalized cropping, not attention.
Why not just scale up a monolithic ViT like VidEoMT? At ViT-L scale VidEoMT is very strong, and the file notes its advantage shrinks with smaller backbones. We compare at matched parameter count and resolution (ViT-S/B against our Swin-T/B) and at 1080p surgical resolution, where global attention cost becomes binding. We concede in advance that at ViT-L on YouTube-VIS, coupling may not hurt. Our claim is restricted to high resolution, small objects and long occlusion.
Proposed Experiments
General benchmarks are OVIS (AP, $AP_{50}$, $AP_{75}$, $AP_{HO}$) and YouTube-VIS 2021 (AP, AR, FPS). We add identity switches, computed by matching predictions to ground-truth tracks per frame, and results bucketed by video length and by instance area. Domain benchmarks are EndoVis 2018 (challenge; mIoU, per-class IoU and boundary F-score for thread, needle and clasper) and Endoscapes2023 (mAP), both run at native resolution.
Three checks decide the outcome. The first is a drift curve: identity-switch rate against frame index on OVIS videos longer than 30 frames. The full model should flatten this curve relative to both InsPro and the stop-gradient-only ablation. If it does not, the first claim fails. The second is a scale curve: mask AP or IoU against instance area. The size-matched window should beat the fixed-level and deformable variants mainly in the smallest third of objects while staying level on large ones, at no more FLOPs. A uniform gain would point to extra capacity rather than scale handling. The third is a resolution sweep from 720p to 1080p on EndoVis. Memory and FPS should stay nearly flat for our model, while the global-attention TAFPNet-style baseline grows with the square of the pixel count. As a probe, we will also track the cosine similarity of each instance's identity sub-query across frames, compared with its similarity to other instances. This checks whether the stop-gradient actually produces a more stable identity code, or whether gains come from elsewhere.
References
- Object Detection for UAV Aerial Scenarios Based on Vectorized IOU
- YOLO-ERF: lightweight object detector for UAV aerial images
- DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
- Decoupled PROB: Decoupled Query Initialization Tasks and Objectness-Class Learning for Open World Object Detection
- InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation
- InstanceFormer: An Online Video Instance Segmentation Framework
- DVIS: Decoupled Video Instance Segmentation Framework
- QueST: Persistent Queries as Semantic Monitors for Drift Suppression in Long-Horizon Tracking
- VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
- Surgical Scene Segmentation by Transformer with Asymmetric Feature Enhancement
- Temporal Propagation of Asymmetric Feature Pyramid for Surgical Scene Segmentation
- Retinal Image Segmentation with Small Datasets
- Optimizing Operating Points for High Performance Lesion Detection and Segmentation Using Lesion Size Reweighting
- Over-and-Under Complete Convolutional RNN for MRI Reconstruction
- A Survey on Vision Transformer
- The Devil Is in the Details: Window-based Attention for Image Compression
- Tracking without bells and whistles
- Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge
- 2018 Robotic Scene Segmentation Challenge