Abstract
ContextRefine-CLIP (CR-CLIP) introduces a bidirectional cross-modal attention flow to enhance video-text retrieval by allowing vision and language features to refine each other dynamically. This architecture addresses the limitations of standard dual-encoder models in capturing fine-grained dependencies in complex egocentric datasets. CR-CLIP secured 1st place in the EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025, achieving a 66.8% mAP and 82.1% nDCG, significantly outperforming previous ensemble-based state-of-the-art solutions with a single, efficient model.
Introduction
Video-text retrieval involves identifying relevant video clips based on textual queries or vice versa. Conventional systems typically employ a dual-encoder architecture, such as CLIP, where vision and text encoders operate independently until a final global comparison. However, this "shallow" interaction is often insufficient for datasets like EPIC-KITCHENS-100, where actions are subtle and "soft labels" are common—multiple descriptions may validly apply to a single clip.
To overcome these challenges, we propose ContextRefine-CLIP (CR-CLIP). By facilitating communication between modalities during the encoding process, CR-CLIP captures the fine-grained dependencies necessary for high-precision retrieval in egocentric environments.
Methodology
CR-CLIP is built upon the AVION framework, incorporating several key innovations:
Cross-Modal Context Refinement (CMCR)
The core of CR-CLIP is the CMCR module, which utilizes bidirectional cross-attention. Text features serve as queries to identify relevant visual components, and visual features simultaneously query the text. This process generates context-aware representations where each modality is refined by the other prior to final matching.
Gated Feed-Forward Network (GatedFFN)
Inspired by modern Transformer architectures, a gated mechanism is applied to the attention output. This GatedFFN enhances the model's capacity to represent complex cross-modal interactions.
Symmetric Multi-Similarity Loss (SMSLoss)
To handle the dataset's soft labels, we employ SMSLoss. This loss function optimizes the relationships between positive and negative pairs based on a correlation matrix, ensuring the model learns nuanced similarities rather than binary classifications.
Enhanced Test-Time Augmentation (TTA)
Robustness is further improved through TTA, including horizontal flipping and multi-scale resizing (87.5%, 100%, and 112.5%). Results from these augmented views are averaged to ensure consistency across different viewpoints and spatial scales.
Experimental Setup
The model was pre-trained on the LLM-augmented Ego4D dataset, a large-scale collection of first-person videos. Fine-tuning and evaluation were performed on the EPIC-KITCHENS-100 Multi-Instance Retrieval task, characterized by its complex, overlapping, and fine-grained first-person kitchen activities.
Results and Discussion
CR-CLIP achieved state-of-the-art performance, securing 1st place on the EPIC-KITCHENS-100 public leaderboard.
As shown in the ablation study (Figure 1), the introduction of the CMCR module significantly boosted performance. Compared to the SMSLoss baseline, the basic CR-CLIP model improved average mAP from 62.1% to 66.2%. The addition of GatedFFN and TTA (Flip+Scale) further pushed the metrics to a peak of 66.8% mAP and 82.1% nDCG.
A key insight is that architectural interaction (CMCR) is more effective than model ensembling. CR-CLIP outperformed the 2024 challenge winner—an ensemble of multiple models—by +3.0% in mAP and +7.8% in nDCG. This demonstrates that allowing text to "guide" visual feature extraction enables the model to distinguish between highly similar actions that global summaries might overlook.
Conclusion
CR-CLIP demonstrates the power of bidirectional contextual refinement in video-text retrieval. While highly effective, future work could explore the scalability of this approach to smaller backbones (e.g., ViT-B) and conduct a more detailed efficiency analysis regarding computational overhead. Additionally, combining multiple CR-CLIP variants through ensembling remains a promising avenue for further performance gains.
Summary updated