Equivariant and Invariant Grounding for Video Question Answering

Yicong Li, Xiang Wang, Junbin Xiao, Tat-Seng Chua

Abstract

Video Question Answering (VideoQA) models often achieve high performance by exploiting spurious correlations rather than genuine visual understanding. This paper introduces EIGV (Equivariant and Invariant Grounding), a framework designed to transform "black-box" VideoQA models into self-interpretable systems. By partitioning video content into causal scenes and environmental contexts, EIGV applies causal principles—specifically equivariance and invariance—to ensure models focus on relevant visual cues. Experimental results across major benchmarks like MSVD-QA, MSRVTT-QA, and NExT-QA demonstrate that EIGV not only improves state-of-the-art accuracy but also provides faithful, intrinsic rationales for its predictions.

Introduction

Most current VideoQA models are opaque, making it difficult to discern if a correct answer stems from actual visual reasoning or dataset bias. For instance, a model might answer "riding a horse" correctly not because it identified the action, but because it associated a green meadow background with that specific activity.

Existing post-hoc explainability methods, such as heatmaps, often lack "faithfulness," showing what humans expect rather than what the model actually utilized. To address this, the authors propose intrinsic interpretability: building the explanation mechanism directly into the model's reasoning process so that the rationale—the specific video clips used—is a fundamental component of the final answer.

Methodology

The authors frame VideoQA through the lens of Causal Theory, partitioning a video into two distinct parts:

  1. Causal Scene ($C$): Frames or objects critical to answering the question.
  2. Environment Scene ($E$): Irrelevant background or context.
Conceptual overview of Causal-Equivariance and Environment-Invariance using a horse-riding example.

The EIGV Framework

The EIGV framework is a model-agnostic "wrapper" consisting of three primary modules added to a standard VideoQA backbone:

  • Grounding Indicator: Employs a Gumbel-Softmax trick to make a "hard" choice, splitting the video into causal and environment segments based on the question.
  • Intervener: The core innovation of the framework. It performs two types of interventions:
  • Causal-Equivariance: It mixes the causal scene and question with another sample. If the input changes semantically (e.g., mixing "girl riding" with "man surfing"), the predicted answer must change proportionally (e.g., "riding + surfing").
  • Environment-Invariance: It swaps the environment scene with random noise or background from other videos. The model is trained to ensure the answer remains unchanged, proving it does not rely on spurious correlations.
  • Disruptor: Utilizes contrastive learning to further sharpen the model's ability to distinguish between a helpful rationale and a misleading one.
The EIGV framework architecture, illustrating the flow from input pair to the final grounded answers.

Experimental Evaluation

The researchers evaluated EIGV on three major benchmarks: MSVD-QA (~50k pairs), MSRVTT-QA (~243k pairs), and NExT-QA (~47.7k pairs). The latter is specifically designed to test causal and temporal reasoning, such as "Why did X happen?", which presents a significant challenge compared to simple descriptive tasks.

Key Results

  • State-of-the-Art Performance: EIGV consistently outperformed existing models across all three datasets, with accuracy gains ranging from +1.2% to +2.3% over previous benchmarks.
  • Backbone Agnosticism: When integrated with existing architectures like Co-Mem (memory-based), HGA (graph-based), or MSPAN (hierarchy-based), EIGV provided significant boosts, including a +5.2% gain in accuracy for specific backbones.
  • Qualitative Grounding: Visualizations confirm that EIGV successfully "zeros in" on correct temporal windows. For example, when queried about a specific action, the model correctly ignores irrelevant segments of the video to focus solely on the causal frames.

Discussion and Future Work

EIGV demonstrates that interpretability and accuracy are not a trade-off; by forcing the model to ignore environmental "noise," it becomes more robust. However, some limitations remain. The current grounding operates at a temporal level (clips); future work could extend this to finer-grained spatial grounding (specific pixels or objects). Additionally, in highly complex scenes with overlapping actions, the model occasionally captures only partial rationales. While the intervention and disruption phases add complexity during training, the framework remains efficient during inference as these modules are removed.

Summary updated

Abstract

Video Question Answering (VideoQA) is the task of answering the natural language questions about a video. Producing an answer requires understanding the interplay across visual scenes in video and linguistic semantics in question. However, most leading VideoQA models work as black boxes, which make the visual-linguistic alignment behind the answering process obscure. Such black-box nature calls for visual explainability that reveals "What part of the video should the model look at to answer the question?". Only a few works present the visual explanations in a post-hoc fashion, which emulates the target model's answering process via an additional method. Instead of post-hoc explainability, we focus on intrinsic interpretability to make the answering process transparent. At its core is grounding the question-critical cues as the causal scene to yield answers, while rolling out the question-irrelevant information as the environment scene. Taking a causal look at VideoQA, we devise a self-interpretable framework, Equivariant and Invariant Grounding for Interpretable VideoQA (EIGV). Specifically, the equivariant grounding encourages the answering to be sensitive to the semantic changes in the causal scene and question; in contrast, the invariant grounding enforces the answering to be insensitive to the changes in the environment scene. By imposing them on the answering process, EIGV is able to distinguish the causal scene from the environment information, and explicitly present the visual-linguistic alignment. Extensive experiments on three benchmark datasets justify the superiority of EIGV in terms of accuracy and visual interpretability over the leading baselines.
BibTeX
@inproceedings{li2022equivariant,
  author = {Yicong Li and Xiang Wang and Junbin Xiao and Tat-Seng Chua},
  title = {{Equivariant and Invariant Grounding for Video Question Answering}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548035},
}
Figure