Hierarchical Banzhaf Interaction for General Video-Language Representation Learning

Peng Jin, Hao Li, Li Yuan, Shuicheng Yan, Jie Chen

TL;DR

This paper introduces Hierarchical Banzhaf Interaction (HBI V2), a novel approach for video-language representation learning that models the interaction between video frames and text words as a multivariate cooperative game. By leveraging Banzhaf Interaction, HBI V2 captures fine-grained semantic correspondences, overcoming limitations of existing coarse-grained methods and achieving state-of-the-art performance across various downstream tasks like retrieval, question answering, and captioning.

What Problem Does This Paper Address?

Current video-language representation learning methods often rely on contrastive learning to align video and text at a global level. While effective, this approach struggles to capture the nuanced, fine-grained interactions between specific video segments and textual phrases. This limitation hinders performance on tasks requiring detailed understanding, such as video question answering and captioning. The paper addresses this by proposing a method to model these fine-grained interactions using cooperative game theory, specifically Banzhaf Interaction, to quantify the incremental benefits of aligning specific video-text pairs.

Methods

The core of HBI V2 lies in its innovative application of cooperative game theory.

  • Hierarchical Banzhaf Interaction (HBI): Video frames and text words are treated as "players" in a cooperative game. Banzhaf Interaction is used to measure the contribution of each player (or coalition of players) to the overall semantic similarity. This is applied hierarchically across "entity," "action," and "event" levels to capture interactions at different granularities.
  • Representation Reconstruction: To mitigate biases in Banzhaf Interaction calculations and improve robustness, the authors propose reconstructing video and text representations by fusing single-modal and cross-modal components. This fusion dynamically balances the granularity of single-modal representations with the adaptive encoding of cross-modal ones.
  • Token Merge Module: To manage the combinatorial explosion of potential coalitions, a token merge module is introduced. This module clusters visual and textual tokens, reducing the number of players and enabling efficient hierarchical interaction. DPC-KNN is used for clustering, followed by a weighted attention mechanism.
  • Encoder-Decoder Framework: For versatility across downstream tasks, HBI V2 employs a flexible encoder-decoder architecture with task-specific prediction heads for VideoQA and video captioning.
  • Training Objective: The training involves a combination of cross-modal contrastive loss, hierarchical Banzhaf Interaction loss, and task-specific losses. Self-distillation across semantic levels is also used to improve generalization.

Datasets

The paper evaluates HBI V2 on a range of established datasets:

  • Text-Video Retrieval: MSRVTT, ActivityNet Captions, DiDeMo
  • Video Question Answering (VideoQA): MSRVTT-QA, MSVD-QA, ActivityNet-QA
  • Video Captioning: MSRVTT

Key Insights and Ideas

  • Game Theory for Fine-Grained Alignment: Applying cooperative game theory, specifically Banzhaf Interaction, offers a principled way to quantify fine-grained semantic correspondences between video and text, moving beyond coarse global alignment.
  • Hierarchical Interaction: Modeling interactions at multiple semantic levels (entity, action, event) is crucial for capturing the complexity of video-language relationships.
  • Representation Fusion for Robustness: Reconstructing representations by blending single-modal and cross-modal features effectively addresses biases in Banzhaf Interaction calculations and improves overall model performance.
  • Interpretability through Interaction: The hierarchical Banzhaf Interaction framework can serve as a visualization tool, offering insights into how the model reasons about cross-modal relationships.

Key Results

HBI V2 demonstrates superior performance across all evaluated tasks:

  • Text-Video Retrieval: Achieves state-of-the-art results on MSRVTT, ActivityNet Captions, and DiDeMo datasets, outperforming previous methods including the prior HBI. For instance, on MSRVTT, HBI V2 achieves R@1 of 49.4 for text-to-video and 47.7 for video-to-text retrieval.
  • VideoQA: Shows significant improvements on MSRVTT-QA, MSVD-QA, and ActivityNet-QA, with HBI V2 achieving an accuracy of 53.4 on MSRVTT-QA.
  • Video Captioning: Achieves competitive performance on MSRVTT, with a CIDEr score of 59.1.
  • Efficiency: The method is efficient, with Banzhaf Interaction being removable during inference, leading to comparable inference times to other advanced methods.

Limitations and Future Work

  • Computational Complexity of Banzhaf Interaction: While the paper uses a proxy model to estimate Banzhaf Interaction for efficiency, the exact calculation is NP-hard. The authors acknowledge this and use an approximation.
  • Dataset Specificity: While HBI V2 performs well across various datasets, the paper notes that improvements on MSRVTT-QA were less pronounced, suggesting that datasets requiring more complex reasoning might pose greater challenges.
  • Future Work: The authors suggest that their framework can be extended to other multimodal tasks and that the visualization capabilities of the hierarchical interaction can be further explored to enhance model interpretability.

Summary updated

BibTeX
@article{jin2025hierarchical,
  author = {Peng Jin and Hao Li and Li Yuan and Shuicheng Yan and Jie Chen},
  title = {{Hierarchical Banzhaf Interaction for General Video-Language Representation Learning}},
  journal = {IEEE TPAMI},
  year = {2025},
  url = {https://openreview.net/forum?id=VjvYRhKIrc},
}
Figure