TL;DR
XTrack is a unified framework that repurposes pretrained video Diffusion Transformers (DiTs) to solve the "cross-view temporal object correspondence" problem. By treating mask prediction as a flow-matching task from RGB latents to mask latents, it jointly models spatial matching and temporal tracking, achieving superior identity preservation and temporal consistency compared to traditional multi-stage pipelines.
What Problem Does This Paper Address?
Identifying the same object across different camera angles (e.g., a first-person "ego" view and a third-person "exo" view) is a fundamental challenge for multi-view scene understanding and embodied AI.
Existing methods typically follow one of two flawed paradigms:
- Track-then-Match: Track the object in the query view, then match each frame independently to the target view. This lacks temporal consistency in the target view.
- Match-then-Track: Find a match in the first frame and then track it independently in both views. If the initial match is slightly off, the error propagates indefinitely.
The core issue is that these methods decouple space (cross-view matching) and time (temporal tracking). They fail to use temporal context from one view to resolve ambiguities (like occlusions or motion blur) in the other.
Methods
The authors introduce XTrack, which leverages the rich spatiotemporal priors already present in large-scale pretrained video DiTs.
- RGB-to-Mask Flow Matching: Instead of the standard diffusion task (noise-to-RGB), XTrack is trained to "transport" RGB latents to object mask latents. This allows the model to focus its capacity on mask inference rather than reconstructing scene structure from scratch.
- Unified Diptych Token Space: Ego and exo video frames are concatenated side-by-side into a "diptych" format. This allows the transformer's self-attention mechanism to naturally perform both within-view tracking and cross-view matching in a single shared token space.
- Grounding-Aware Enhancements: To prevent the model from getting confused by semantically similar "distractor" objects, the authors add:
- Focal Attention Alignment: A loss function that supervises cross-view attention maps using foreground masks.
- Hard Negative Conditioning: Training the model to distinguish the target object from other similar objects in the same scene.
- Local Context Crops: A high-resolution "zoom-in" branch for small objects that preserves spatial positioning via Rotary Positional Embeddings (RoPE).
Datasets
- Ego-Exo4D: The primary benchmark. However, it only provides sparse annotations (1 FPS).
- Pseudo-Label Curation Pipeline: To enable effective video-level training, the authors developed a pipeline to convert sparse labels into dense ones. It uses bidirectional propagation (via SAM2), cross-view geometric verification (via VGGT features), and semantic filtering (via DINOv3) to ensure high-quality supervision.
Key Insights and Ideas
- Generative Models as Perception Backbones: Pretrained video generators are essentially "world simulators." They already understand that objects persist even when occluded or viewed from different angles. Repurposing these priors is more efficient than training a tracking model from scratch.
- Joint Reasoning is Key: By processing both views simultaneously in one transformer, the model can use "evidence" from a clear exo-view to resolve uncertainty in a cluttered or occluded ego-view.
- Attention Alignment Matters: Standard DiTs often attend to semantically similar objects (e.g., any "cup" in the scene). Explicitly supervising the attention maps (Focal Attention Alignment) is crucial for instance-level accuracy.
Key Results
XTrack demonstrates a significant leap in temporal consistency and identity preservation:
- Temporal Consistency: On the
mR@IoUmetric (which measures tracking retention), XTrack achieves 0.954 (Ego2Exo), significantly higher than frame-wise matching baselines (~0.6-0.8). - Visibility Accuracy (VA): It excels at predicting when an object disappears or reappears, scoring 0.900 compared to the ~0.4-0.6 range of prior "Track-then-Match" methods.
- Qualitative Superiority: XTrack maintains a stable mask even during rapid camera motion or when the object is partially truncated, scenarios where previous methods typically fail.
Limitations and Future Work
- Small Objects: Despite the local crop strategy, very small objects can still be lost due to the heavy spatial compression in the DiT's latent space.
- Context Length: The model is limited by the finite context window of the pretrained DiT. Long-range consistency across minutes of video remains a challenge.
- Inference Speed: As a diffusion-based model, XTrack requires multiple denoising steps, making it slower than pure feed-forward segmentation networks. Future work may involve distilling the model into a faster feed-forward version.
Summary updated