Encoding and Controlling Global Semantics for Long-form Video Question Answering

Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong-Duy T. Nguyen, See-Kiong Ng, Anh Tuan Luu

Abstract

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence ($C^3$) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets.

The Challenge of Long-form Video Reasoning

Video Question Answering (videoQA) has seen significant progress in short-clip scenarios. However, scaling these systems to videos spanning minutes or hours presents a massive computational challenge. Most current models rely on sampling or selection strategies—picking a small subset of frames or regions to process.

This approach creates a significant "gap" in global semantics. If a question requires understanding a character's overarching motivation or a sequence of events distributed across a feature-length film, models that only view isolated frames often fail to grasp the broader context. While standard Transformers could theoretically process more frames, their quadratic ($O(L^2)$) complexity makes them prohibitively expensive for long-form content.

The GSMT Framework

To bridge this gap, we propose the Gated State space Multi-modal Transformer (GSMT). This architecture introduces two primary innovations designed to capture global context efficiently and accurately.

1. Gated State Space Layer (SSL)

Instead of relying solely on standard attention for global encoding, GSMT utilizes a State Space Layer (SSL). SSLs are mathematically optimized to handle extremely long sequences with linear complexity, making them far more suitable for long-form video than traditional Transformers.

To prevent the model from being overwhelmed by irrelevant background data, we incorporate a gating mechanism into the SSL. This unit acts as a filter, controlling the flow of global information into the final visual representations and ensuring that only pertinent context is retained.

The GSMT architecture featuring the Gated State Space Layer and the C3 objective.

2. Cross-modal Compositional Congruence ($C^3$)

The $C^3$ objective is a novel training loss designed to align visual attention with the logic of the query. It measures the "congruence" between the model's visual patch selection and its understanding of the entities described in the text. This ensures that the global information captured is directly relevant to the specific question being asked, focusing on relationships between entities rather than just simple object identification.

New Frontiers: Ego-QA and MAD-QA

To rigorously test these capabilities, we introduce two massive new benchmarks that push the boundaries of video length:

  • Ego-QA: Focused on first-person (egocentric) videos with an average length of 17.5 minutes.
  • MAD-QA: Based on full-length movies, with an average video length of 1.9 hours.

These datasets are significantly more demanding than previous benchmarks. For instance, answering a question in Ego-QA requires watching an average of 1,204 seconds of video, compared to just 100 seconds for the popular EgoSchema benchmark.

Experimental Results

GSMT was evaluated across seven datasets, including the five standard benchmarks AGQA-v2, NExT-QA, STAR, Env-QA, and EgoSchema.

  • State-of-the-Art Performance: GSMT consistently outperformed existing models like MIST and CoVGT. On AGQA-v2, it achieved a 1.77% accuracy improvement over the previous best.
  • Long-form Superiority: On the new Ego-QA and MAD-QA datasets, GSMT surpassed the second-best models by 2.67% and 1.96% respectively.
  • Qualitative Success: The model demonstrated particular strength in answering "Why" and "How" questions, which require understanding complex chains of events over time.
Qualitative examples comparing GSMT and MIST-CLIP on the MAD-QA and Ego-QA benchmarks.

Conclusion and Future Directions

By integrating Gated SSLs and the $C^3$ objective, GSMT proves that it is possible to maintain global context in long-form video without sacrificing computational efficiency. However, a significant gap remains between AI and human performance, particularly on the MAD-QA movie benchmark (19% vs. 73% accuracy). Future work will focus on improving generalizability across diverse video domains and incorporating multicultural contexts to ensure these systems are robust and globally applicable.

Summary updated

Abstract

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence (C^3) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets.
BibTeX
@misc{nguyen2024encoding,
  author = {Thong Thanh Nguyen and Zhiyuan Hu and Xiaobao Wu and Cong-Duy T. Nguyen and See-Kiong Ng and Anh Tuan Luu},
  title = {{Encoding and Controlling Global Semantics for Long-form Video Question Answering}},
  year = {2024},
  eprint = {2405.19723},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.19723},
}
Figure