In traditional computer vision, tracking objects across a video was a two-step process: first, detect objects in every frame independently, and then use a separate algorithm—like a Kalman filter or Hungarian matching—to stitch those detections into trajectories. This "detect-then-match" paradigm is brittle, often failing when objects overlap or momentarily disappear.
A new research direction replaces these hand-crafted association heuristics with Temporal Query Propagation. In this framework, the Transformer's "object queries"—the high-dimensional vectors used to find objects in a single image—are no longer static. Instead, they become persistent, evolving "slots" that follow an object through time. By passing these queries from one frame to the next, the model treats tracking not as a post-processing step, but as an inherent part of the neural network's internal representation.
From Static Slots to Evolving Hypotheses
In a standard Transformer detector like DETR, queries are learned "slots" that look for objects in specific spatial regions. To extend this to video, researchers like Tim Meinhardt and Alexander Kirillov introduced TrackFormer, which uses autoregressive embeddings. The output embeddings from frame $t$ are fed back into the decoder as queries for frame $t+1$.
The intuition is that if a query successfully "captured" a car in the first frame, that query's high-dimensional state already contains the car's appearance and location. By using it as the starting point for the next frame, the model doesn't have to "find" the car again; it simply refines the car's updated position. This turns the query into a persistent identity. As Fangao Zeng and Xiangyu Zhang demonstrate in MOTR, these "track queries" evolve over time, treating tracking as a continuous set-prediction problem rather than a discrete matching task.
Queries as Working Memory
This propagation mechanism acts as a form of "working memory." In InstanceFormer, Rajat Koner and colleagues use a temporal memory queue to propagate instance-specific queries and spatial reference points. This provides a strong inductive bias: an object at coordinates $(x, y)$ in frame 1 is likely to be near those coordinates in frame 2.
By propagating these queries, the model maintains identity through "implicit association." Because the same query index is used to decode the object across the entire sequence, the model naturally knows that "Query #5" is the same person in every frame. This eliminates the need for external re-identification (Re-ID) modules. Fei He and the team behind InsPro show that this "query-proposal" propagation allows for real-time performance (up to 45 FPS) because the model avoids the computational overhead of complex matching algorithms.
Robustness through Temporal Denoising
A major challenge in this direction is training the model to handle occlusions. If an object disappears for five frames, the query must "wait" and maintain its state until the object reappears.
To solve this, Shuxiao Ding introduced Temporal Query Denoising (TQD) in TQD-Track. During training, the model is given ground-truth queries from previous frames that have been intentionally corrupted with noise. The model must learn to "denoise" these queries to find the object's current state. This forces the network to learn robust motion patterns and appearance models, ensuring that the query doesn't "drift" or lose the object during complex interactions. Similarly, Zhuofan Zong proposed Historical Object Prediction, which requires the model to predict where objects were in the past, further strengthening the temporal link between queries.
Beyond Tracking: Interaction and Mapping
The power of persistent queries extends beyond simple bounding boxes. In video generation, Siyoon Jin and Seungryong Kim developed MATRIX, which aligns internal attention maps with object mask tracks. By ensuring that the model's "attention" follows the same query-based path, they prevent common video generation artifacts like "identity drift," where a person's face might change mid-video.
In the realm of autonomous driving, Jiacheng Chen and Hang Ma applied this to HD mapping with MapTracker. Instead of detecting lane lines in every frame and trying to stitch them together, they propagate "map queries" that represent road elements. This ensures that the map remains stable as the car moves, treating the static environment as a series of "tracked" entities.
By treating object identity as a persistent, evolving vector, these researchers have moved computer vision away from "flickering" frame-by-frame detections toward a more stable, "video-first" understanding of the world.