In video tasks like video instance segmentation (VIS), video object detection (VOD), and multiple object tracking (MOT), the traditional pipeline follows a "detect-then-associate" paradigm. For instance, Bergmann et al. in Tracking Without Bells and Whistles demonstrated that a standard object detector could track objects by reusing bounding boxes from a previous frame as proposals for the next. However, relying on hand-crafted data association algorithms (such as Hungarian matching) or complex post-processing heuristics often introduces computational bottlenecks and fails under severe occlusion or motion blur.
The advent of Transformer-based detectors introduced object queries—learnable vectors that probe image feature maps to decode specific object instances. If a single query vector can learn to find an object in a static image, can it maintain that object's identity across a sequence of video frames?
This insight forms the foundation of temporal query propagation. Instead of treating tracking as a separate post-processing step, these methods allow tracking to emerge naturally by carrying query representations across time.
Queries as Persistent Identity Slots
To understand why this works, consider what an object query contains. It is a high-level representation encoding both semantic content (what the object looks like) and spatial context (where the object is located).
In a video Transformer, if we assign a fixed query slot to an object, that slot can track the object implicitly across frames. As reviewed by Li et al. in Transformer-Based Visual Segmentation: A Survey, reusing the same query vectors across consecutive frames naturally handles object association without explicit tracking modules.
Early offline models leveraged this by processing entire video clips at once. For example, Wang et al. in End-to-End Video Instance Segmentation With Transformers (VisTR) used a fixed set of learned queries to decode instances across a sequence of frames. Because the instance predicted at a specific query index remains consistent across the temporal dimension, tracking becomes a direct byproduct of the architecture.
Similarly, Botach et al. in End-to-End Referring Video Object Segmentation with Multimodal Transformers (MTTR) and Wu et al. in Language as Queries for Referring Video Object Segmentation (ReferFormer) applied this concept to multimodal tasks, ensuring temporal consistency by reusing query indices across frames to track referred objects.
Offline Paradigm (Clip-level):
[Frame 1, Frame 2, Frame 3] ---> [Transformer Decoder] ---> Fixed Query Slots (Implicit Tracking)
From Fixed Slots to Recurrent Query Evolution
While fixed query slots work well for short video clips, they struggle with long, streaming, or online video sequences where new objects enter and old ones leave. To solve this, researchers shifted from static query slots to recurrent query propagation.
Instead of keeping queries static across a clip, the model feeds the output tokens or decoded queries from frame $t$ as the input queries for frame $t+1$. This creates an auto-regressive chain where queries evolve dynamically alongside the objects they track.
Online Paradigm (Frame-by-frame):
Frame t ---> [Decoder] ---> Output Query (t)
|
v (Propagated as Input)
Frame t+1 ---> [Decoder] ---> Output Query (t+1)
Qiu et al. in PSVT: End-to-End Multi-Person 3D Pose and Shape Estimation with Progressive Video Transformers utilized this recurrent structure to initialize queries using output tokens from preceding frames, allowing the model to progressively track human instances over time.
In the domain of unsupervised 3D tracking, Sautier et al. in UNIT: Unsupervised Online Instance Segmentation through Time demonstrated that feeding queries from one time step into the next eliminates the need for Hungarian matching during inference entirely.
For one-shot localization, Kumar 0004 et al. in QDETRv: Query-Guided DETR for One-Shot Object Localization in Videos passed decoded object queries between consecutive frames to maintain a persistent memory of a target object across the sequence, stabilizing localization.
Anchoring Queries with Spatial Proposals
A pure content-based query can get lost if an object moves rapidly or is temporarily occluded, as its appearance features change. To anchor the queries geographically within the frame, modern architectures propagate pairs of instance queries and spatial proposals (such as bounding boxes or reference points).
He et al. in InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation proposed reusing both the updated query and the spatial proposal from the previous frame as the initialization for the current frame. This dual propagation provides a strong inductive bias: the model knows both what it is tracking (via the query embedding) and where it was last seen (via the spatial proposal).
Koner et al. in InstanceFormer: An Online Video Instance Segmentation Framework similarly propagated instance-specific queries, spatial reference points, and classification scores, allowing the model to track objects through gradual changes in appearance and position without re-initializing detection at every step.
In OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation, Wu et al. propagated the highest-scoring object query along with its prediction box and positional embeddings to provide a robust spatial-semantic prior for the next time step.
Why It Works: Sparse vs. Dense Alignment
Traditional video models often rely on dense pixel-level alignment, such as computing optical flow or cross-frame feature warping, which is computationally expensive. Temporal query propagation works efficiently because it shifts the alignment from dense pixel space to a sparse, high-level embedding space.
Dense Alignment (Traditional):
[Frame t Pixels] <--- Optical Flow / Warping ---> [Frame t+1 Pixels] (Expensive)
Sparse Alignment (Query Propagation):
[Object Query t] ------------------------------> [Object Query t+1] (Efficient)
As shown by He et al. in QueryProp: Object Query Propagation for High-Performance Video Object Detection, high-level object queries serve as compact proxies for temporal information. These queries interact directly with new frame features, bypassing the need for explicit spatial warping.
Similarly, Zhou et al. in TransVOD: End-to-End Video Object Detection With Spatial-Temporal Transformers linked sparse object query embeddings across frames to maintain temporal consistency instead of using dense pixel-level optical flow.
This sparse interaction drastically reduces latency, enabling real-time performance on edge devices. For example, Gao et al. in MobileInst: Video Instance Segmentation on the Mobile demonstrated that temporal query propagation internalizes tracking capabilities within the query embeddings, ensuring stable performance during inference without increasing latency.
Key Foundational Work
End-to-End Referring Video Object Segmentation with Multimodal Transformers (2021) — contributed 2 concept elements to this direction.
A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformer (2021) — contributed 2 concept elements to this direction.
Multiple Object Tracking With Correlation Learning (2021) — contributed 2 concept elements to this direction.