Hierarchical Banzhaf Interaction for General Video-Language Representation Learning

Peng Jin, Hao Li, Li Yuan, Shuicheng Yan, Jie Chen

TL;DR

HBI V2 is a novel framework that models fine-grained video-text alignment as a cooperative game, treating video frames and text words as individual players. By leveraging game-theoretic Banzhaf Interactions and a dynamic representation reconstruction module, it achieves state-of-the-art performance across video retrieval, video question answering, and video captioning without requiring dense, element-level annotations.

What Problem Does This Problem Address?

Most existing video-language models rely on contrastive learning to align videos and text. However, because datasets rarely contain dense, element-level annotations (e.g., mapping the specific word "pizza" to the exact video frame containing a pizza), these models typically settle for coarse-grained, global alignment.

This lack of fine-grained interaction hinders performance on complex downstream tasks like Video Question Answering (VideoQA) and video captioning, which demand a detailed understanding of visual entities and textual phrases. While cross-modal joint encoding can help filter out redundant video information, it often introduces calculation biases or destroys the fine granularity of single-modal representations when the text and video content diverge.

Fig. 1: (a) Previous methods only learn a global semantic in- teraction from the coarse-grained labels of video-text pairs. (b) We model multimodal alignment as a cooperative game process, utilizing Banzhaf Interaction to evaluate possible correspondence between video frames and text words.

Methods

The authors introduce Hierarchical Banzhaf Interaction V2 (HBI V2), which reframes multimodal learning through the lens of multivariate cooperative game theory:

  1. Video-Text as Game Players: Video frames and text words are formulated as players in a cooperative game where their mutual semantic similarity serves as the payoff function.
  2. Banzhaf Interaction: To evaluate how well elements correspond, the model uses the Banzhaf Interaction index. This metric quantifies the additional semantic benefit a specific coalition of players (e.g., a frame and a word) brings to the table compared to the costs of lost coalitions with other players.
  3. Representation Reconstruction: To eliminate calculation bias caused by redundant video data, HBI V2 dynamically fuses single-modal and cross-modal representations using a learnable gating factor ($\gamma$ and $\delta$). This preserves the fine granularity of single-modal features while retaining the text-conditioned adaptability of cross-modal features.
  4. Token Merge Module: Computing interactions for every possible combination of frames and words is computationally impossible ($2^{N_v+N_t}$). The authors use a density-peaks clustering algorithm (DPC-KNN) to hierarchically merge tokens into entity-level, action-level, and event-level coalitions.
  5. Flexible Encoder-Decoder Framework: Task-specific prediction heads are appended to the core encoder, allowing a single architecture to seamlessly adapt to retrieval, QA, and caption generation.
Fig. 2: The intuition of employing Banzhaf Interaction in video-language representation learning. When certain players (frames and words) form a coalition, it entails the exclusion of these players from potential coalitions with others, rendering them mutually exclusive from the target coalition. Banzhaf Interaction quantifies the disparity between the benefits derived from the coalition and the costs incurred due to the lost coalitions. Therefore, Banzhaf Interaction effectively captures the incremental benefits conferred by the coalition. We refer the reader to Eq. 1 for the detailed formula.

Datasets

The model was evaluated across seven widely recognized benchmarks:

  • Text-Video Retrieval: MSRVTT (10K YouTube videos), ActivityNet Captions (20K videos), and DiDeMo (10K videos).
  • Video Question Answering: MSRVTT-QA (243K QA pairs), MSVD-QA (50.5K QA pairs), and ActivityNet-QA (58K complex web videos).
  • Video Captioning: MSRVTT-Caption.

No new datasets were introduced in this work.

Key Insights and Ideas

  • Alignment as Active Cooperation: Instead of treating alignment as a passive matrix multiplication of similarities, framing it as a cooperative game measures how much a specific combination of visual and textual elements actively drives the overall match.
  • The Granularity vs. Adaptability Trade-off: Purely cross-modal representations are excellent at text-conditioned filtering but collapse frame-level details. Blending single-modal and cross-modal representations dynamically provides the best of both worlds.
  • Hierarchical Distillation: Entity-level interactions (low-level semantics) converge first during training. The authors found that distilling this early entity-level knowledge down to action and event levels significantly guides and stabilizes the learning of higher-level semantic features.

Key Results

HBI V2 consistently outperformed both its predecessor (HBI V1) and task-specific state-of-the-art methods:

  • Text-Video Retrieval: On MSRVTT, it achieved a Text-to-Video R@1 of 49.4%, outperforming HBI V1 (48.6%) and Prompt Switch (47.8%).
  • VideoQA: It achieved an answer accuracy of 53.4% on MSVD-QA, a notable 1.3% improvement over the previous HBI baseline.
  • Video Captioning: It reached a CIDEr score of 59.1 on MSRVTT, surpassing competitive task-specific models like SwinBERT (55.9).
  • Inference Efficiency: Crucially, because the Banzhaf Interaction module is only used as a proxy training objective, it can be completely removed during inference. HBI V2 adds a mere 1 second of processing time to the entire test set compared to standard baselines, making it highly practical.

Limitations and Future Work

The authors noted that the performance gains on the MSRVTT-QA dataset were less pronounced than on other benchmarks. They attribute this to HBI V2's primary focus on video-language alignment, whereas MSRVTT-QA presents a steeper challenge rooted in complex video-language reasoning.

Additionally, calculating the exact Banzhaf Interaction is an NP-hard problem. While the authors successfully bypassed this during training by pre-training a tiny proxy neural network to estimate the values using Mean Squared Error (MSE) loss, developing more mathematically exact yet scalable calculation methods remains an open area for future research.

Summary updated

Abstract

Multimodal representation learning, with contrastive learning, plays an important role in the artificial intelligence domain. As an important subfield, video-language representation learning focuses on learning representations using global semantic interactions between pre-defined video-text pairs. However, to enhance and refine such coarse-grained global interactions, more detailed interactions are necessary for fine-grained multimodal learning. In this study, we introduce a new approach that models video-text as game players using multivariate cooperative game theory to handle uncertainty during fine-grained semantic interactions with diverse granularity, flexible combination, and vague intensity. Specifically, we design the Hierarchical Banzhaf Interaction to simulate the fine-grained correspondence between video clips and textual words from hierarchical perspectives. Furthermore, to mitigate the bias in calculations within Banzhaf Interaction, we propose reconstructing the representation through a fusion of single-modal and cross-modal components. This reconstructed representation ensures fine granularity comparable to that of the single-modal representation, while also preserving the adaptive encoding characteristics of cross-modal representation. Additionally, we extend our original structure into a flexible encoder-decoder framework, enabling the model to adapt to various downstream tasks. Extensive experiments on commonly used text-video retrieval, video-question answering, and video captioning benchmarks, with superior performance, validate the effectiveness and generalization of our method.
BibTeX
@article{jin2025hierarchical,
  author = {Peng Jin and Hao Li and Li Yuan and Shuicheng Yan and Jie Chen},
  title = {{Hierarchical Banzhaf Interaction for General Video-Language Representation Learning}},
  journal = {IEEE TPAMI},
  year = {2025},
  url = {https://openreview.net/forum?id=VjvYRhKIrc},
}
Figure