Learning Semantic Alignment with Global Modality Reconstruction for Video-Language Pre-training towards Retrieval

Mingchao Li, Xiaoming Shi, Haitao Leng, Wei Zhou, Hai-Tao Zheng, Kuncai ZhangAAAI 2023 · DOI

Introduction

Video-language pre-training (VLP) aims to establish robust links between visual scenes and textual descriptions. While current models excel at "token-level" alignment—matching specific words like "soldier" to corresponding visual objects—they often struggle with sequence-level semantic alignment. This limitation prevents models from capturing the global narrative and logical flow of a video sequence.

The Challenges: Context and Noise

The authors identify two primary hurdles in existing VLP approaches:

  1. Missing the Global Context: Models frequently focus on isolated keywords while ignoring sequence logic. For instance, a model might identify a "person" and the action "laying down" but fail to distinguish between different subjects if the full sequence context isn't integrated.
  2. Semantic Over-alignment: Real-world video data is inherently noisy. A narrator might discuss unrelated topics while performing a task, such as mentioning "battled chickens" while fixing a lawnmower. Previous models attempted to force-align every visual element with every spoken word, leading to "over-alignment" where modality-specific noise (like background objects) confuses the shared semantic representation.
Figure 1: Real-world video data often contains modality-specific noise, such as a lawnmower being visually present while the narration discusses unrelated topics.

The FEEL Framework

To address these issues, the researchers developed FEEL (For lEarning sEmantic aLignments). This framework shifts the focus from local matching to understanding the global meaning of entire sequences through two novel mechanisms built upon a standard cross-modal Transformer.

Figure 6: The FEEL architecture featuring the Modal Converting Block (MCB) for global modality reconstruction.

Global Modality Reconstruction (GMR)

Instead of simple feature matching, FEEL employs a Modality Converting Block (MCB) to reconstruct the entire representation of one modality from the other. For example, the model attempts to generate a video embedding sequence based solely on text features. This process forces the model to internalize the global, sequential flow of information rather than relying on isolated snapshots.

Cross-Modal Self-Contrasting (CSC)

To mitigate over-alignment, FEEL uses a triplet loss mechanism called Cross-Modal Self-Contrasting. By converting a feature twice (e.g., Text $\rightarrow$ Video $\rightarrow$ Text), the model learns that the "twice-converted" version should remain close to the original. The first conversion acts as a semantic filter, stripping away modality-specific noise (like background details not mentioned in text) and preserving only the shared semantic core.

Experimental Results

FEEL was pre-trained on a 7.56 million clip subset of HowTo100M and evaluated across several benchmarks, including MSR-VTT, TVR, and DiDeMo.

  • Video Retrieval: On MSR-VTT, FEEL outperformed the previous state-of-the-art (TACO) by +0.7% in R@1 and +2.1% in R@10. On TVR, it achieved 31.15% R@1, surpassing the HERO model.
  • Moment Localization: FEEL showed significant gains in finding specific clips within a larger corpus, outperforming HERO by +8.37% in R@100 on the TVR validation set.
  • Ablation Studies: The importance of sequence-level learning was confirmed when removing the GMR component led to a massive performance drop (over 6% in R@1 on MSR-VTT).

Conclusion

By prioritizing global sequence reconstruction over local token matching, FEEL provides a more robust solution for video-language understanding. The framework's ability to treat different modalities as semantic filters allows it to ignore irrelevant noise, leading to state-of-the-art performance in both retrieval and localization tasks. Future work may involve extending this architecture to include additional modalities like audio and depth information.

Summary updated

BibTeX
@inproceedings{li2023learning,
  author = {Mingchao Li and Xiaoming Shi and Haitao Leng and Wei Zhou and Hai-Tao Zheng and Kuncai Zhang},
  title = {{Learning Semantic Alignment with Global Modality Reconstruction for Video-Language Pre-training towards Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25222},
}
Figure