In the world of computer vision, Video Referring Expression Comprehension (REC) is the high-stakes version of "Where’s Waldo?" Instead of a static image, you are given a video and a natural language description (e.g., "A gray elephant walks from left to right"), and the model must draw a bounding box around that specific elephant in every frame.
While image-based REC has seen massive leaps, video REC remains a "flickery" mess. A new paper from researchers at Peking University and Columbia University, titled Correspondence Matters for Video Referring Expression Comprehension, introduces DCNet to solve exactly that.
TL;DR
DCNet (Dual Correspondence Network) stabilizes video object grounding by explicitly learning two types of associations: inter-frame (tracking the same object across time) and cross-modal (linking specific words to specific visual patches). By "mining" these correspondences through feature similarity, the model achieves state-of-the-art results and can be used as a "plug-and-play" enhancement for existing architectures.
What Problem Does This Paper Address?
Current state-of-the-art (SOTA) models for video REC suffer from two primary headaches:
- Inconsistent Localization: Because videos involve motion blur, lighting changes, and occlusions, a model might correctly identify an elephant in Frame 1, lose it in Frame 2 when it passes behind a tree, and then accidentally pick a different elephant in Frame 3.
- Referent Confusion: Sentences often mention multiple objects (e.g., "A rabbit is running around a deer"). Models frequently get "distracted" by the contextual object (the deer) and fail to focus on the actual referent (the rabbit).
The authors argue that these failures happen because models treat frames too much like isolated images and lack a fine-grained map between words and pixels.
Methods
The authors propose the Dual Correspondence Network (DCNet). The "Dual" refers to two novel contrastive learning losses that don't require any extra manual annotations:
1. Inter-frame Contrastive Learning ($\mathcal{L}_{inter}$)
Instead of just looking at the annotated bounding box (sparse alignment), DCNet performs dense alignment. It calculates the cosine similarity between every patch in one frame and every patch in another. It "mines" the most similar patches across frames to serve as positive pairs for contrastive learning. This forces the model to learn features that are temporally consistent, helping it "remember" the object even during occlusions or blur.
2. Cross-modal Contrastive Learning ($\mathcal{L}_{cross}$)
To stop the model from confusing the "rabbit" with the "deer," DCNet builds a patch-word alignment. It computes the similarity between visual patches and individual word embeddings from the query. By selecting the most responsive patch-word pairs as positive samples, the model learns to associate the word "rabbit" with the actual visual features of a rabbit, rather than just looking at the global meaning of the sentence.
The "Plug-and-Play" Factor: These losses are architecture-agnostic. You can take an existing video REC model, add these two loss functions during training, and see an immediate performance boost without changing the underlying inference logic.
Datasets
The researchers evaluated DCNet on three major benchmarks:
- VID-Sentence: A large-scale dataset with 7,654 trimmed videos and natural language captions.
- Lingual OTB99 (LiOTB): Based on the OTB100 tracking dataset, containing 100 videos.
- RefCOCO: A classic image REC dataset used to prove that the cross-modal components also improve static image grounding.
Key Insights and Ideas
- Dense > Sparse: Aligning every spatial patch across frames is significantly more robust than just aligning the features within the ground-truth bounding boxes.
- Similarity Mining Works: You don't need "gold standard" labels for word-to-patch correspondence. Simply using cosine similarity to find the "best fits" during training is enough to supervise the model effectively.
- Addressing "Empty" Words: The authors found that "patch-topk" (finding the best words for a patch) works better than "word-topk." This is because many words in a query (like "a," "the," or "of") have no visual counterpart, and forcing them to align with patches introduces noise.
Key Results
DCNet set new SOTA records across the board:
- VID-Sentence: Achieved 65.54% [email protected], outperforming the previous SOTA (Co-grounding) by 2.19%.
- Generalization: When added to the existing Co-grounding model, DCNet's losses boosted performance by 1.48% absolute improvement.
- RefCOCO: Even on static images, DCNet outperformed established models like MAttN by up to 4.72%, proving the strength of its cross-modal alignment.
- Qualitative Stability: Visualizations show that DCNet produces much more stable "confidence maps" across video frames compared to baseline models, which often show erratic jumps in attention.
Limitations and Future Work
While DCNet is powerful, the authors noted a trade-off in temporal aggregation. They tested an "adjacent strategy" (comparing only neighboring frames) versus a "fully-connected strategy" (comparing all frames). While the fully-connected version was slightly more accurate, it was computationally expensive ($O(T^2)$).
Future work could explore more efficient ways to handle long-range temporal dependencies or integrate these correspondence losses into even larger transformer-based multimodal pre-training frameworks.
Check out the code: https://github.com/mengcaopku/DCNet
Summary updated