ContextRefine-CLIP for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025

Jing He, Yiqing Wang, Lingling Li, Kexin Zhang, Puhua Chen

Abstract

ContextRefine-CLIP (CR-CLIP) introduces a bidirectional cross-modal attention flow to enhance video-text retrieval by allowing vision and language features to refine each other dynamically. This architecture addresses the limitations of standard dual-encoder models in capturing fine-grained dependencies in complex egocentric datasets. CR-CLIP secured 1st place in the EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025, achieving a 66.8% mAP and 82.1% nDCG, significantly outperforming previous ensemble-based state-of-the-art solutions with a single, efficient model.

Introduction

Video-text retrieval involves identifying relevant video clips based on textual queries or vice versa. Conventional systems typically employ a dual-encoder architecture, such as CLIP, where vision and text encoders operate independently until a final global comparison. However, this "shallow" interaction is often insufficient for datasets like EPIC-KITCHENS-100, where actions are subtle and "soft labels" are common—multiple descriptions may validly apply to a single clip.

To overcome these challenges, we propose ContextRefine-CLIP (CR-CLIP). By facilitating communication between modalities during the encoding process, CR-CLIP captures the fine-grained dependencies necessary for high-precision retrieval in egocentric environments.

Methodology

CR-CLIP is built upon the AVION framework, incorporating several key innovations:

Cross-Modal Context Refinement (CMCR)

The core of CR-CLIP is the CMCR module, which utilizes bidirectional cross-attention. Text features serve as queries to identify relevant visual components, and visual features simultaneously query the text. This process generates context-aware representations where each modality is refined by the other prior to final matching.

Gated Feed-Forward Network (GatedFFN)

Inspired by modern Transformer architectures, a gated mechanism is applied to the attention output. This GatedFFN enhances the model's capacity to represent complex cross-modal interactions.

Symmetric Multi-Similarity Loss (SMSLoss)

To handle the dataset's soft labels, we employ SMSLoss. This loss function optimizes the relationships between positive and negative pairs based on a correlation matrix, ensuring the model learns nuanced similarities rather than binary classifications.

Enhanced Test-Time Augmentation (TTA)

Robustness is further improved through TTA, including horizontal flipping and multi-scale resizing (87.5%, 100%, and 112.5%). Results from these augmented views are averaged to ensure consistency across different viewpoints and spatial scales.

Experimental Setup

The model was pre-trained on the LLM-augmented Ego4D dataset, a large-scale collection of first-person videos. Fine-tuning and evaluation were performed on the EPIC-KITCHENS-100 Multi-Instance Retrieval task, characterized by its complex, overlapping, and fine-grained first-person kitchen activities.

Results and Discussion

CR-CLIP achieved state-of-the-art performance, securing 1st place on the EPIC-KITCHENS-100 public leaderboard.

Performance comparison and ablation study of CR-CLIP on the EPIC-KITCHENS-100 Multi-Instance Retrieval task.

As shown in the ablation study (Figure 1), the introduction of the CMCR module significantly boosted performance. Compared to the SMSLoss baseline, the basic CR-CLIP model improved average mAP from 62.1% to 66.2%. The addition of GatedFFN and TTA (Flip+Scale) further pushed the metrics to a peak of 66.8% mAP and 82.1% nDCG.

A key insight is that architectural interaction (CMCR) is more effective than model ensembling. CR-CLIP outperformed the 2024 challenge winner—an ensemble of multiple models—by +3.0% in mAP and +7.8% in nDCG. This demonstrates that allowing text to "guide" visual feature extraction enables the model to distinguish between highly similar actions that global summaries might overlook.

Conclusion

CR-CLIP demonstrates the power of bidirectional contextual refinement in video-text retrieval. While highly effective, future work could explore the scalability of this approach to smaller backbones (e.g., ViT-B) and conduct a more detailed efficiency analysis regarding computational overhead. Additionally, combining multiple CR-CLIP variants through ensembling remains a promising avenue for further performance gains.

Summary updated

Abstract

This report presents ContextRefine-CLIP (CR-CLIP), an efficient model for visual-textual multi-instance retrieval tasks. The approach is based on the dual-encoder AVION, on which we introduce a cross-modal attention flow module to achieve bidirectional dynamic interaction and refinement between visual and textual features to generate more context-aware joint representations. For soft-label relevance matrices provided in tasks such as EPIC-KITCHENS-100, CR-CLIP can work with Symmetric Multi-Similarity Loss to achieve more accurate semantic alignment and optimization using the refined features. Without using ensemble learning, the CR-CLIP model achieves 66.78mAP and 82.08nDCG on the EPIC-KITCHENS-100 public leaderboard, which significantly outperforms the baseline model and fully validates its effectiveness in cross-modal retrieval. The code will be released open-source on https://github.com/delCayr/ContextRefine-Clip
BibTeX
@misc{he2025contextrefine,
  author = {Jing He and Yiqing Wang and Lingling Li and Kexin Zhang and Puhua Chen},
  title = {{ContextRefine-CLIP for EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2025}},
  year = {2025},
  eprint = {2506.10550},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.10550},
}
Figure