TL;DR
This paper tackles the challenge of video moment retrieval (VMR), where models struggle with dataset biases related to temporal location. They introduce a causality-inspired framework that disentangles moment content from its location and uses causal intervention to remove spurious correlations, leading to more robust and generalizable VMR models.
What Problem Does This Paper Address?
Video Moment Retrieval (VMR) aims to find a specific segment in a video that matches a given text query. While current methods have made progress, they often rely on dataset biases, particularly the temporal location of moments, rather than truly understanding the video content and query. This leads to models that perform well on training data but fail to generalize to new, unseen data distributions. The authors identify two key temporal location biases: a "long-tailed annotation" bias where moments are often annotated at the beginning or end of videos, and a bias where certain query types (e.g., "open") are strongly correlated with specific temporal locations. These biases act as "confounders," obscuring the true relationship between video content and query, and hindering robust VMR.
Methods
The proposed solution, Deconfounded Cross-modal Matching (DCM), employs a two-pronged approach rooted in causal inference:
- Moment Representation Disentangling: The core idea is to separate the visual content of a moment from its temporal location. This is achieved by training a network to disentangle a moment's representation into two independent latent vectors: one for content and one for location. This disentanglement is guided by two loss functions: a reconstruction loss that ensures the learned location feature matches the original temporal information, and an independence loss that forces the content and location features to be statistically independent. This step aims to remove the "L→V" link in the causal graph, where location influences the moment representation.
- Causal Intervention (Backdoor Adjustment): To address the "L→Y" link (location influencing the prediction), DCM applies causal intervention. Using do-calculus, they modify the model's prediction to be based on the do-calculus intervention, effectively forcing the model to consider all possible temporal locations for a moment, rather than relying on the biased distribution seen during training. This is approximated by averaging the model's output over all possible disentangled location features. This intervention aims to remove the spurious correlation between the moment and the prediction that is mediated by location.
Additionally, the authors introduce counterfactual training by including negative query-video pairs to teach the model what would happen if the target moment didn't exist, further encouraging focus on content. They also propose Out-of-Distribution (OOD) testing to better evaluate generalization by altering the temporal location distribution of moments in the test set without affecting training.
Datasets
The proposed DCM method is evaluated on three standard VMR datasets:
- ActivityNet-Captions (ANet-Cap): A large dataset with diverse human activities.
- Charades-STA: A popular dataset focused on indoor activities.
- DiDeMo: A dataset featuring human activities in real-world scenarios.
The authors also introduce an Out-of-Distribution (OOD) testing methodology, which involves inserting random video clips at the beginning of test videos to shift the temporal location distribution, thereby testing model robustness.
Key Insights and Ideas
- Temporal Location as a Confounder: The paper convincingly argues that temporal location is not just a feature but a "confounder" that distorts the true relationship between video content and query in VMR.
- Disentanglement is Crucial: Separating visual content from temporal location is essential for building models that truly understand the video, not just memorizing common annotation patterns.
- Causal Intervention for Robustness: Applying causal intervention, specifically backdoor adjustment, is key to making VMR models generalize better by forcing them to consider all temporal possibilities, not just the biased ones.
- OOD Testing is Necessary: Standard IID evaluation is insufficient for VMR due to location biases. OOD testing reveals the true generalization capabilities of models.
- Dataset Biases are Pervasive: The paper provides strong empirical evidence that existing VMR models heavily exploit temporal location biases, leading to poor performance when these biases are altered.
Key Results
- Significant Improvement: DCM consistently improves baseline models (CMI and TCN) on both IID and OOD testing sets across all three datasets.
- Stronger OOD Performance: Improvements are often larger on OOD sets, demonstrating DCM's effectiveness in enhancing generalization. For example, relative improvements on R@1(IoU>0.5) for OOD testing on ANet-Cap were 23.68%.
- New State-of-the-Art: DCM-based methods achieve new SOTA results on R@1(IoU>0.7) for IID testing on ANet-Cap, Charades-STA, and DiDeMo.
- Ablation Studies Confirm Contributions: Removing any component of DCM (disentangling, independence loss, causal intervention, counterfactual loss) leads to performance degradation, especially on OOD tests, validating the importance of each part.
- Qualitative Evidence: Visual examples show DCM's robustness to temporal distribution shifts, while baseline models fail.
Limitations and Future Work
- Limitations:
- The paper notes that while DCM retains the "good" aspects of location bias (e.g., for temporal language queries), it might slightly underperform on very long moments in IID settings where location bias is highly predictive.
- The OOD evaluation on ANet-Cap was adjusted by excluding long moments, as they were too easily predicted by location bias alone.
- Future Work:
- Extend intervention strategies to the user query to better understand complex user intents.
- Apply the deconfounding approach to other location-sensitive tasks like visual grounding and temporal activity localization.
- Further explore the integration of DCM with existing VMR methods to enhance their generalizability.
Summary updated