Introduction
Video-language pre-training (VLP) aims to establish robust links between visual scenes and textual descriptions. While current models excel at "token-level" alignment—matching specific words like "soldier" to corresponding visual objects—they often struggle with sequence-level semantic alignment. This limitation prevents models from capturing the global narrative and logical flow of a video sequence.
The Challenges: Context and Noise
The authors identify two primary hurdles in existing VLP approaches:
- Missing the Global Context: Models frequently focus on isolated keywords while ignoring sequence logic. For instance, a model might identify a "person" and the action "laying down" but fail to distinguish between different subjects if the full sequence context isn't integrated.
- Semantic Over-alignment: Real-world video data is inherently noisy. A narrator might discuss unrelated topics while performing a task, such as mentioning "battled chickens" while fixing a lawnmower. Previous models attempted to force-align every visual element with every spoken word, leading to "over-alignment" where modality-specific noise (like background objects) confuses the shared semantic representation.
The FEEL Framework
To address these issues, the researchers developed FEEL (For lEarning sEmantic aLignments). This framework shifts the focus from local matching to understanding the global meaning of entire sequences through two novel mechanisms built upon a standard cross-modal Transformer.
Global Modality Reconstruction (GMR)
Instead of simple feature matching, FEEL employs a Modality Converting Block (MCB) to reconstruct the entire representation of one modality from the other. For example, the model attempts to generate a video embedding sequence based solely on text features. This process forces the model to internalize the global, sequential flow of information rather than relying on isolated snapshots.
Cross-Modal Self-Contrasting (CSC)
To mitigate over-alignment, FEEL uses a triplet loss mechanism called Cross-Modal Self-Contrasting. By converting a feature twice (e.g., Text $\rightarrow$ Video $\rightarrow$ Text), the model learns that the "twice-converted" version should remain close to the original. The first conversion acts as a semantic filter, stripping away modality-specific noise (like background details not mentioned in text) and preserving only the shared semantic core.
Experimental Results
FEEL was pre-trained on a 7.56 million clip subset of HowTo100M and evaluated across several benchmarks, including MSR-VTT, TVR, and DiDeMo.
- Video Retrieval: On MSR-VTT, FEEL outperformed the previous state-of-the-art (TACO) by +0.7% in R@1 and +2.1% in R@10. On TVR, it achieved 31.15% R@1, surpassing the HERO model.
- Moment Localization: FEEL showed significant gains in finding specific clips within a larger corpus, outperforming HERO by +8.37% in R@100 on the TVR validation set.
- Ablation Studies: The importance of sequence-level learning was confirmed when removing the GMR component led to a massive performance drop (over 6% in R@1 on MSR-VTT).
Conclusion
By prioritizing global sequence reconstruction over local token matching, FEEL provides a more robust solution for video-language understanding. The framework's ability to treat different modalities as semantic filters allows it to ignore irrelevant noise, leading to state-of-the-art performance in both retrieval and localization tasks. Future work may involve extending this architecture to include additional modalities like audio and depth information.
Summary updated