MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling

Jiaqi Xu, Bo Liu, Yunkuo Chen, Mengli Cheng, Xing ShiAAAI 2024 · DOI

TL;DR

This paper introduces MuLTI, a novel framework for video-and-language understanding that tackles the challenge of efficiently processing long video sequences and bridging the gap between pre-training and downstream tasks. MuLTI achieves state-of-the-art performance by employing a "Text-Guided MultiWay-Sampler" for efficient feature fusion and a new "Multiple Choice Modeling" pre-training task.

What Problem Does This Paper Address?

Video-and-language understanding tasks, such as video question answering (videoQA) and text-video retrieval, are crucial for many real-world applications. However, existing methods often struggle with two main challenges:

  1. Computational Cost and Long Sequences: Many current models use heavy multi-modal encoders and feature fusion modules that are computationally expensive. This becomes particularly problematic when dealing with long video sequences or extensive text, leading to performance degradation or requiring significant computational resources. Existing approaches to condense video features, like mean pooling or random sampling, can lead to the loss of critical information.
  2. Domain Gap Between Pre-training and Downstream Tasks: While pre-training on large datasets improves performance, there's often a significant gap between the pre-training objectives and the specific requirements of downstream tasks like videoQA. Existing pre-training tasks don't always effectively align video and text representations for these specific applications, making it difficult to transfer learned knowledge.

Methods

MuLTI addresses these challenges with two key innovations:

  • Text-Guided MultiWay-Sampler: This is the core of MuLTI's efficient feature fusion. Instead of relying on generic sampling or simple pooling, this sampler uses text features to guide the sampling of video features. It incorporates an "Adapt-Pooling" mechanism that learns the importance of different video sequence blocks, allowing for selective condensation of features. This approach aims to preserve crucial information while reducing computational load, enabling the model to handle longer sequences more effectively. The sampler shares self-attention modules but uses distinct feed-forward networks for video and text to handle multi-modal features efficiently.
  • Multiple Choice Modeling (MCM): To bridge the pre-training and downstream task gap, especially for videoQA, the authors introduce MCM. This novel pre-training task frames the problem as a multiple-choice question. The model is presented with a video and several text descriptions, and it must select the description that best matches the video. This task is designed to mimic the cognitive process of videoQA, thereby improving the model's ability to align video and text features and enhancing its cross-modal reasoning capabilities. MCM can be seamlessly integrated with other standard pre-training tasks like Masked Language Modeling (MLM), Video Text Matching (VTM), and Video Text Comparison (VTC) without requiring additional manual annotations.

Datasets

MuLTI is trained and evaluated on a variety of datasets:

  • Pre-training: WebVid-2M (2.5 million video-text pairs) and CC-3M (3 million image-text pairs) are used for pre-training.
  • Downstream Tasks:
    • Video Question Answering (VideoQA): MSRQ (MSRVTT-QA), MSVQ (MSVD-QA), and TGIF-QA (including TGIF-Action, TGIF-Transition, and TGIF-Frame).
  • Text-Video Retrieval: MSRR (MSRVTT-Ret) and DiDeMo (DiDeMo-Ret).
  • Multi-Label Classification: A custom dataset derived from an online advertising app, containing 486k videos with captions and 21696 labels.

Key Insights and Ideas

  • Text as a Guide for Video Understanding: The core idea that text can effectively guide the sampling and condensation of video features is a significant insight. This leverages the inherent conciseness and semantic richness of text to make video processing more efficient and effective.
  • Adapt-Pooling for Information Preservation: The Adapt-Pooling mechanism is crucial for preventing information loss during feature condensation. By learning feature importance, it ensures that the most relevant parts of the video are retained, which is vital for complex tasks.
  • MCM as a Bridge for VideoQA: The Multiple Choice Modeling pre-training task is a clever way to directly address the domain gap for videoQA. By mimicking the question-answering format, it forces the model to learn more relevant alignments between video and text, leading to better performance on this specific task.
  • Scalability and Efficiency: The paper demonstrates that MuLTI can be scaled to different model sizes (MuLTI-S, MuLTI-B, MuLTI-L) and maintains efficiency even with longer text inputs, as shown in memory usage comparisons.

Key Results

MuLTI achieves state-of-the-art performance across multiple benchmarks:

  • VideoQA: MuLTI surpasses existing baseline models on MSRQ, MSVQ, and TGIF datasets.
  • Text-Video Retrieval: MuLTI shows highly competitive results on MSRVTT and DiDeMo, particularly excelling in DiDeMo.
  • Multi-Label Classification: MuLTI significantly outperforms VIOLET and ALPRO, even when compared without OCR transcripts.
  • Efficiency: MuLTI demonstrates significantly lower memory costs compared to methods like ALPRO and VIOLET, especially as the number of frames increases. For instance, with 16 frames and 512 text length, MuLTI-B uses 10551 MB, while ALPRO uses 21819 MB.

Limitations and Future Work

  • MVM Task Conflict: The paper notes that the Masked Frame Modeling (MVM) pre-training task can conflict with other tasks, leading to performance degradation. They chose not to include MVM in their final pre-training setup.
  • Frozen Layers and Overfitting: While unfreezing more layers generally improves performance, there's a risk of overfitting due to excessive parameter adjustments. The paper suggests using adapters in shallow layers to mitigate this.
  • Future Work: The authors implicitly suggest further exploration of adapter mechanisms and potentially more sophisticated ways to integrate different pre-training tasks. The success of MCM also opens avenues for designing more task-specific pre-training objectives.

Summary updated

BibTeX
@inproceedings{xu2024multi,
  author = {Jiaqi Xu and Bo Liu and Yunkuo Chen and Mengli Cheng and Xing Shi},
  title = {{MuLTI: Efficient Video-and-Language Understanding with Text-Guided MultiWay-Sampler and Multiple Choice Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28448},
}
Figure