Exploiting Textual Semantics for Robust Cross-View Object Correspondence

Bing Fan, Yunhe Feng, Yan Huang, Heng Fan

TL;DR

TeSCo (Textual Semantics for Correspondence) is a new framework that solves the difficult problem of tracking the same object across different camera angles (e.g., a head-mounted camera vs. a room camera). By generating and refining textual descriptions of an object, TeSCo creates a "viewpoint-invariant" anchor that helps vision models stay locked on a target even when its visual appearance changes drastically.

What Problem Does This Paper Address?

In computer vision, Cross-View Object Correspondence (CVOC) is the task of identifying the same object in two different video streams—typically an "ego-centric" (first-person) view and an "exo-centric" (third-person) view.

This is incredibly hard because an object looks completely different from different angles. A basketball might look like a giant orange sphere to the person holding it, but just a tiny dot to a camera mounted on the ceiling. Traditional methods rely purely on visual matching, which often fails when lighting changes, objects are partially hidden, or the perspective shifts too much. The authors argue that while pixels change, the concept of the object remains stable. A "red basketball with a white stripe" is a description that holds true regardless of which camera is looking at it.

Methods

TeSCo is built on top of the state-of-the-art SAM 2 (Segment Anything Model 2) and introduces a three-step pipeline to integrate language into the tracking process:

  1. Textual Description Generation: Since text isn't provided in standard tracking tasks, TeSCo uses a vision-language model (specifically DAM) to look at the query object and generate a detailed description (e.g., "An orange basketball made of textured leather with a distinct white stripe").
  2. Iterative Textual Feature Refinement (ITFR): Not every part of a description is visible from every angle. The ITFR module progressively adjusts the textual features by comparing them to the "search view" (the target camera). This ensures the model focuses on textual cues that are actually helpful for the current frame.
  3. Text-Conditioned Cross-Modality Fusion (TCF): This is the "secret sauce" that merges vision and language. Instead of just adding the text data to the image data, TCF uses the text to modulate the visual features. It acts like a smart filter, highlighting parts of the image that match the description (reweighting) and shifting the feature distribution to be more robust.
Figure 1: Overview of the proposed TeSCo that exploits rich textual information for robust cross- view correspondence through iterative textual feature refinement (ITFR) and text-conditioned cross- modality fusion (TCF) modules. Best viewed in pdf for all figures throughout the paper.

Datasets

The authors evaluated TeSCo on two major benchmarks:

  • Ego-Exo4D: A massive dataset featuring synchronized first-person and third-person videos of real-world activities like cooking, sports, and repair work.
  • HANDAL-X: A dataset focused on real-world manipulable objects with complex poses, used here to test how well the model generalizes to new objects without specific training (zero-shot learning).

Key Insights and Ideas

  • Language as an Anchor: High-level semantic attributes (category, color, material) are more stable across viewpoints than raw pixel values. Language acts as a bridge that visual-only models lack.
  • Refinement is Mandatory: You can't just "inject" text and expect it to work. The ITFR module (shown below) is crucial because it filters out parts of the description that might be occluded or ambiguous in the second camera's view.
  • Modulation over Concatenation: The TCF module proves that using text to dynamically scale and shift visual features is far more effective than simply "stacking" them together.
Figure 2: Illustration of the ITFR module.

Key Results

TeSCo achieved a new state-of-the-art performance across the board:

  • Ego-Exo4D: Achieved a mean Intersection over Union (mIoU) of 65.35%, a massive 9.85% absolute gain over the SAM 2 baseline (55.50%).
  • HANDAL-X: In zero-shot tests (testing on objects the model hadn't seen during training), TeSCo reached 85.10% IoU, outperforming the next best model by over 6%.
  • Qualitative Impact: Visualizations show that with textual semantics, the model's "attention" is much more concentrated on the actual target, whereas visual-only models often get distracted by similar-looking objects in the background.
Figure 4: Image (a): query-view frames; image (b): search-view frames; image (c): attention maps in search view with textual semantics; and (d) attention maps in search view without textual semantics. The visualized attention maps is obtained by aggregating the attention maps from the two attention modules in TeSCo. We can observe that, the attention map with textual semantics can more accurately focus on the target region, evidencing the effectiveness of textual information.

Limitations and Future Work

Despite its success, TeSCo has two main hurdles:

  1. Frame-level Dependency: It generates descriptions frame-by-frame. If a frame is blurry or the object is momentarily hidden, the description quality drops. The authors want to explore "temporal" descriptions that look at the whole video clip for better stability.
  2. Computational Cost: Running a vision-language model to generate text adds significant overhead. Future work will focus on making the text generation and extraction process more "resource-friendly" for real-time applications.

Summary updated

Abstract

Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code is released at https://github.com/VAIL-Fan/TeSCo.
BibTeX
@misc{fan2026exploiting,
  author = {Bing Fan and Yunhe Feng and Yan Huang and Heng Fan},
  title = {{Exploiting Textual Semantics for Robust Cross-View Object Correspondence}},
  year = {2026},
  eprint = {2611.02356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02356},
}
Figure