Abstract
Current video retrieval (VR) models are often evaluated on benchmarks that may not fully reflect true linguistic understanding. This study investigates the extent to which state-of-the-art VR models utilize compositionality and syntax. By applying systematic perturbations to text queries across object, action, and syntactic axes, we reveal that most models operate as "bag-of-words" systems, prioritizing objects over actions and showing surprising robustness to word-order changes. Notably, CLIP-based architectures demonstrate superior compositional awareness compared to models trained exclusively on video-text pairs.
Introduction
Video Retrieval (VR) involves finding a specific video based on a text query. A high-performing model should ideally understand compositionality—how objects, attributes, and actions are structured by syntax to convey meaning. However, many current evaluation benchmarks are noisy, allowing models to succeed by matching keywords (e.g., "car") without understanding the underlying relationships (e.g., "the man is driving the red car"). This paper investigates whether current models actually understand these complex relationships or merely act as sophisticated keyword matchers.
Methodology
The authors developed a comprehensive test bed to stress-test 12 state-of-the-art video retrieval models, categorized into two groups: those trained directly on video-text pairs (e.g., Frozen-in-Time, VIOLET) and those adapted from CLIP's image-text representations (e.g., CLIP4Clip, XCLIP).
To measure semantic depth, 10 types of perturbations were applied to text captions across three axes:
- Objects & Attributes: Removing elements, shifting attributes between objects, or replacing them with random words.
- Actions: Removing, negating (adding "not"), or replacing verbs.
- Syntactics: Removing filler words, shuffling word order, or reversing sentences.
Performance was evaluated on standard benchmarks: MSRVTT, MSVD, and DiDeMo.
Results and Analysis
The Dominance of Objects
Objects and attributes are the most critical components for current models. As shown in Figure 1, replacing objects or providing only partial object information leads to a precipitous drop in Recall@1 (R@1) scores. In many cases, performance craters from approximately 40% to below 10%, indicating that models rely heavily on these spatial anchors to identify the correct video.
Syntactic Indifference and the "Bag-of-Words" Reality
Conversely, models appear remarkably indifferent to syntax. Shuffling or reversing a sentence only caused a minor performance dip (approx. 5-9%). This suggests that models are not "reading" the sentence structure but are instead scanning for keywords. Figure 2 illustrates this trend, where the "Shuffle" and "Reverse" perturbations result in scores that remain close to the original performance levels.
The CLIP Advantage and Action Blindness
Models built on CLIP showed significantly better syntactic and compositional awareness than those trained solely on video datasets. This suggests that the massive scale of image-text pre-training provides a stronger semantic foundation. However, all models struggled with action negation; adding "not" to a verb barely changed model behavior, indicating a failure to understand the absence or reversal of an action.
Conclusion
Current video retrieval models are heavily biased toward spatial information (objects) rather than temporal sequences (actions) or linguistic structure (syntax). Future research should focus on architectures that move beyond single-frame bias to capture temporal dynamics and training regimes that explicitly penalize models for ignoring word order and attribute-object bindings.
Summary updated