Video Referring Expression Comprehension (REC) is a challenging task in computer vision that requires a model to localize a specific object across all frames of a video based on a natural language description. Unlike static image REC, video REC must handle temporal dynamics such as motion blur, occlusions, and lighting changes. Current state-of-the-art models often struggle with temporal inconsistency—where the model "loses" the object between frames—and referent confusion, where the model fails to distinguish the target object from other entities mentioned in the query.
Researchers from Peking University and Columbia University address these issues in their paper, Correspondence Matters for Video Referring Expression Comprehension, by introducing the Dual Correspondence Network (DCNet). This framework stabilizes grounding by explicitly learning inter-frame and cross-modal associations through feature similarity mining.
The Dual Correspondence Network (DCNet)
DCNet introduces two novel contrastive learning losses that can be integrated into existing video REC architectures as a "plug-and-play" enhancement. These losses do not require additional manual annotations, as they rely on mining correspondences from the existing feature space.
1. Inter-frame Correspondence ($\mathcal{L}_{inter}$)
To combat inconsistent localization, DCNet performs dense alignment across frames. Rather than simply aligning features within ground-truth bounding boxes, the model calculates cosine similarity between every spatial patch in one frame and every patch in another. By mining the most similar patches to serve as positive pairs for contrastive learning, the model learns temporally consistent features. This allows the network to maintain a stable "memory" of the object even when it is partially obscured or blurred by motion.
2. Cross-modal Correspondence ($\mathcal{L}_{cross}$)
To resolve referent confusion (e.g., distinguishing a "rabbit" from a "deer" when both are mentioned), DCNet implements patch-word alignment. The model computes similarity between visual patches and individual word embeddings from the query. By selecting the most responsive patch-word pairs, the model learns to associate specific visual features with their corresponding linguistic terms. The authors found that a "patch-to-top-k-words" strategy is most effective, as it avoids forcing alignment with non-visual function words like "the" or "of."
Experimental Evaluation
The researchers evaluated DCNet on three major benchmarks: VID-Sentence, Lingual OTB99 (LiOTB), and RefCOCO.
On the VID-Sentence dataset, DCNet achieved a state-of-the-art 65.54% [email protected], a 2.19% improvement over previous methods. Furthermore, when the DCNet losses were added to existing baseline models, they provided a consistent performance boost (e.g., +1.48% for the Co-grounding model), demonstrating the framework's versatility. Even on static images (RefCOCO), the cross-modal alignment component allowed DCNet to outperform established models like MAttN by up to 4.72%.
Conclusion
DCNet demonstrates that explicit correspondence mining is a powerful tool for stabilizing video object grounding. By enforcing dense alignment across time and between modalities, the model achieves superior robustness against common video artifacts and linguistic ambiguity. The architecture-agnostic nature of the $\mathcal{L}_{inter}$ and $\mathcal{L}_{cross}$ losses makes DCNet a highly practical solution for improving current and future video REC systems.
Summary updated