For years, the dominant paradigm in visual object tracking was the Siamese network. These architectures utilized two separate, parallel neural network branches: one to extract features from a target template (the object to track) and another to extract features from a larger search region (the current video frame). Only at the very end of the pipeline would these features meet, typically fused via a cross-correlation or cross-attention layer to localize the target.
While intuitive, this decoupled approach suffers from a fundamental flaw: feature extraction is entirely target-agnostic. The network processes the search region blindly, without knowing what it is looking for until the final fusion step.
The one-stream transformer tracking paradigm completely upends this design. Instead of separating feature extraction and relation modeling into sequential steps, one-stream architectures unify them into a single pass through a shared Transformer backbone.
The Mechanics of One-Stream Fusion
The core mechanism behind one-stream tracking is elegant in its simplicity: token concatenation.
Instead of passing the target template and the search region through separate network branches, the images are broken down into patches, flattened, and projected into visual tokens. Before entering the Transformer backbone, the template tokens ($T$) and search region tokens ($S$) are concatenated into a single, unified sequence:
$$\text{Input} = [T_1, T_2, \dots, T_n, S_1, S_2, \dots, S_m]$$
As Boyu Chen et al. demonstrated with Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking (SimTrack), this simple structural change eliminates the need for complex, hand-crafted post-processing or specialized cross-correlation heads. A standard, single-branch Vision Transformer (ViT) can handle both feature extraction and target matching simultaneously.
Two-Stream (Siamese):
Template ----> [ Encoder ] ----\
===> [ Cross-Correlation ] ===> Prediction
Search ----> [ Encoder ] ----/
One-Stream:
Template ---\
===> [ Concatenation ] ===> [ Shared Transformer Encoder ] ===> Prediction
Search ---/
Why It Works: Native Self-Attention as a Matchmaker
To understand why this unified approach outperforms traditional architectures, we must look at how information flows through a Transformer's self-attention mechanism.
In a standard self-attention layer, every token in the input sequence computes an attention score with every other token. When the input sequence contains both template and search tokens, the resulting attention matrix naturally splits into four quadrants:
- Template-to-Template ($T \to T$): Extracts intrinsic appearance features of the target.
- Search-to-Search ($S \to S$): Extracts contextual and spatial features of the environment.
- Template-to-Search ($T \to S$) & Search-to-Template ($S \to T$): Performs cross-image relation modeling and feature matching.
As Heng Fan et al. observed in SwinTrack: A Simple and Strong Baseline for Transformer Tracking, this concatenation allows symmetric, deep interactions within shared transformer blocks. Rather than waiting until the final layer to compare the template and search region, a one-stream tracker executes relation modeling at every single layer of the network.
The Power of Target-Aware Features
The primary advantage of this deep interaction is the generation of target-aware features.
In a two-stream network, the search branch must retain all visual information because it does not know which object is relevant. This often leads to the loss of subtle, critical target details during aggressive downsampling layers, or causes the model to get distracted by background clutter.
In a one-stream architecture, because the search tokens attend to the template tokens from layer one, the network dynamically tunes its internal representations to the specific object being tracked throughout the entire depth of the model. As Qiangqiang Wu et al. showed in Scalable Video Object Segmentation with Simplified Framework (SimVOS), this native self-attention allows the network to filter out background noise early on and focus its capacity entirely on target-relevant features.
This early information exchange is particularly vital in challenging environments. For instance, Yan Miao et al. leveraged this property in Coordinate-Aware Thermal Infrared Tracking Via Natural Language Modeling to successfully localize targets in low-contrast thermal imagery, where background and foreground signatures are highly ambiguous.
Computational Efficiency and Edge Deployment
Intuitively, concatenating sequences increases the sequence length, which scales the computational complexity of self-attention quadratically ($O(N^2)$). However, because a one-stream architecture completely removes the need for a separate fusion network and duplicate encoder branches, the overall pipeline is often significantly more streamlined and faster in practice.
Furthermore, the unified nature of the stream allows for elegant, dynamic optimization strategies. Because the network develops an early understanding of where the target is located within the search region, it can actively discard irrelevant background tokens in later layers.
The authors of Adaptive and Background-Aware Vision Transformer for Real-Time UAV Tracking (Aba-ViTrack) utilized this by introducing an adaptive token-halting mechanism. By dropping uninformative background tokens mid-network, they achieved state-of-the-art precision while maintaining blistering speeds (~180 FPS on desktop GPUs and over 35 FPS on embedded edge hardware), proving that one-stream tracking is highly viable for resource-constrained robotics and UAV applications.
Generalization Beyond 2D Bounding Boxes
The elegance of treating feature extraction and matching as a single sequence-modeling problem means the one-stream paradigm generalizes remarkably well across diverse computer vision domains:
- 3D Object Tracking: Teli Ma et al. introduced Synchronize Feature Extracting and Matching: A Single Branch Framework for 3D Object Tracking (SyncTrack), demonstrating that concatenating 3D template and search point clouds into a single-branch Transformer prevents the severe information loss that typically occurs when downsampling sparse LiDAR data in isolated branches.
- Class-Agnostic Counting: In Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting, Zhicheng Wang et al. discovered that standard, unmodified Vision Transformers can count arbitrary objects simply by processing query images and exemplar patches as a single concatenated sequence, relying entirely on native self-attention to find correspondences.
- Sequence-to-Sequence Tracking: Taking the sequence paradigm to its logical conclusion, Xin Chen et al. developed SeqTrack: Sequence to Sequence Learning for Visual Object Tracking. By utilizing a unified encoder to process the template and search regions simultaneously, they paired it with a decoder that directly "reads out" bounding box coordinates as a series of discrete tokens, eliminating task-specific regression heads entirely.
By shifting the tracking philosophy from "extract features independently, then fuse" to "interact immediately, extract concurrently," the one-stream transformer paradigm provides a simpler, faster, and fundamentally more accurate foundation for visual relationship modeling.
Key Foundational Work
MixFormer: End-to-End Tracking with Iterative Mixed Attention (2022) — contributed 4 concept elements to this direction.
Adaptive and Background-Aware Vision Transformer for Real-Time UAV Tracking (2023) — contributed 2 concept elements to this direction.
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022) — contributed 2 concept elements to this direction.