Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentation

Suhwan Cho, Seunghoon Lee, Minhyeok Lee, Jungho Lee, Sangyoun LeearXiv 2025 · DOI · arXiv

Abstract

Referring Video Object Segmentation (RVOS) traditionally relies on entangled fusion strategies that process textual and visual features simultaneously across all video frames. However, this approach often leads to target ambiguity and temporal inconsistency. We present FindTrack, a novel framework that decouples the RVOS task into two distinct stages: target identification and temporal propagation. By identifying the target object in a single high-confidence "key frame" and subsequently tracking it using a dedicated propagation module, FindTrack achieves state-of-the-art performance on major benchmarks, including Ref-YouTube-VOS, Ref-DAVIS17, and MeViS, while significantly improving robustness in complex scenarios.

Introduction

The goal of Referring Video Object Segmentation (RVOS) is to segment a specific object throughout a video sequence based on a natural language description (e.g., "the girl in a green raincoat"). Current state-of-the-art methods typically employ an "entangled" architecture where language features are fused with visual features at every frame. While intuitive, this strategy suffers from two primary limitations. First, it introduces ambiguity; in scenes with multiple similar objects, the model may switch targets between frames if the textual alignment is not perfectly consistent. Second, it lacks temporal coherence, as the model essentially re-identifies the object in every frame rather than leveraging the inherent continuity of video data.

The FindTrack Framework

FindTrack addresses these issues by introducing a "Find First, Track Next" paradigm. This approach treats the initial identification as a high-precision search and the subsequent segmentation as a robust tracking problem.

1. Target Identification (Find First)

Instead of processing the entire video, FindTrack samples a set of candidate frames. For each candidate, it performs per-frame segmentation using a pre-trained model (EVF-SAM). To select the optimal starting point, the system calculates a Mask Score based on:

  • Segmentation Confidence: The internal certainty of the mask's spatial boundaries.
  • Vision-Text Alignment: Utilizing Alpha-CLIP to verify that the pixels within the mask align semantically with the language prompt.

The frame with the highest score is designated as the Key Frame, serving as a stable anchor for the entire sequence.

2. Temporal Propagation (Track Next)

Once the target is localized in the key frame, the task transitions from language-based identification to pixel-based tracking. FindTrack employs a memory-based tracking network, Cutie, to propagate the mask bi-directionally from the key frame. This ensures that the object is tracked based on its visual appearance and motion, maintaining consistency even during occlusions or when the language prompt becomes less descriptive for specific frames.

Figure 4: FindTrack successfully identifies and segments a specific target among similar-looking individuals in a complex outdoor scene.

Experimental Results

FindTrack was evaluated on three primary RVOS benchmarks, demonstrating superior performance across the board. The model achieved a $\mathcal{J}\&\mathcal{F}$ score of 73.7 on Ref-YouTube-VOS and 74.2 on Ref-DAVIS17, representing a significant 6.2% improvement over previous leading methods. On the MeViS dataset, which focuses on long-form, motion-heavy videos, it reached a score of 53.2.

Qualitative analysis reveals that FindTrack is significantly more resilient to "distractor" objects than entangled models. By anchoring the segmentation to a high-confidence key frame, the model avoids the "identity switching" common in previous architectures.

Figure 6: The model maintains a stable mask on a specific player during a dynamic basketball game, demonstrating robustness to rapid motion and crowded environments.

Discussion and Conclusion

The success of FindTrack underscores the value of decoupling identification from propagation. This separation not only improves accuracy but also allows the model to leverage specialized, off-the-shelf components for each sub-task. Remarkably, FindTrack can operate in a zero-shot capacity, though fine-tuning (FindTrack++) yields the best results.

However, the framework is not without limitations. The current inference speed of approximately 10 fps is constrained by the initial identification stage. Furthermore, the system's overall success is heavily dependent on the quality of the key frame; a failure in the "Find First" stage can lead to error propagation. Future work will focus on optimizing the candidate sampling process and integrating more advanced vision-language models to further enhance the robustness of the initial identification.

Summary updated

Abstract

Referring video object segmentation aims to segment and track a target object in a video using a natural language prompt. Existing methods typically fuse visual and textual features in a highly entangled manner, processing multi-modal information together to generate per-frame masks. However, this approach often struggles with ambiguous target identification, particularly in scenes with multiple similar objects, and fails to ensure consistent mask propagation across frames. To address these limitations, we introduce FindTrack, an efficient decoupled framework that separates target identification from mask propagation. FindTrack first adaptively selects a key frame by balancing segmentation confidence and vision-text alignment, establishing a robust reference for the target object. This reference is then utilized by a dedicated propagation module to track and segment the object across the entire video. By decoupling these processes, FindTrack effectively reduces ambiguities in target association and enhances segmentation consistency. FindTrack significantly outperforms all existing methods on public benchmarks, demonstrating its superiority.
BibTeX
@misc{cho2025find,
  author = {Suhwan Cho and Seunghoon Lee and Minhyeok Lee and Jungho Lee and Sangyoun Lee},
  title = {{Find First, Track Next: Decoupling Identification and Propagation in Referring Video Object Segmentation}},
  year = {2025},
  eprint = {2503.03492},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03492},
}
Figure