Modeling Event-level Causal Representation for Video Classification

Yuqing Wang, Lei Meng, Haokai Ma, Haibei Huang, Xiangxu Meng

TL;DR

This paper introduces ECRL, a novel approach to video classification that moves beyond object-centric features to model event-level causal relationships. By employing causal inference techniques, ECRL effectively disentangles background and data biases, leading to more accurate classification of complex events.

What Problem Does This Paper Address?

Current video classification methods often rely on extracting features from individual frames and aggregating them. While effective for many tasks, this approach struggles with classifying videos that describe events, like "fixing a bicycle," because it tends to focus on static object information within frames rather than the dynamic interplay of actions and their temporal dependencies. This leads to spurious correlations between background elements or superficial visual cues and the actual event, hindering accurate classification. The paper aims to bridge this gap by developing a representation that captures the causal structure of events within videos.

Methods

ECRL is built upon two core modules:

  • Frame-to-Video Causal Modeling (F2VCM): This module constructs a video-level causal graph. It first extracts foreground and background information from each frame. Then, it models both in-frame object interactions and cross-frame correlations between foreground and background elements to build a comprehensive causal graph.
  • Causality-aware Event-level Representation Inference (CERI): This module addresses biases identified in the causal graph. It employs two key causal interventions:
  • Back-door Intervention (Background Debiasing - BD): This technique aims to remove spurious correlations caused by background confounders. It uses a "confounder dictionary" derived from visual context to de-bias the model's understanding of background influences.
  • Front-door Intervention (Global-Local Causal Attention - GLCA): This method tackles data biases by using a mediator to capture event-level visual information. It employs a global-local attention mechanism to focus on relevant frames and features, effectively filtering out noise.

By combining these modules, ECRL learns event-level causal representations that are less susceptible to superficial correlations.

Datasets

The paper evaluates ECRL on two widely-used video classification datasets:

  • MSR-VTT: A dataset for video description, used here for video classification.
  • ActivityNet: A large-scale dataset for human activity understanding.

The authors provide statistics for training and testing splits for both datasets.

Key Insights and Ideas

  • Beyond Object-Centricity: The core idea is that true event understanding in videos requires moving beyond just recognizing objects in frames. It necessitates understanding the causal relationships and temporal dynamics that define an event.
  • Causality as a Solution to Bias: The paper highlights how traditional data-driven methods are prone to background and data biases. Causal inference provides a principled framework to identify and mitigate these biases, leading to more robust representations.
  • Deconstructing Video Bias: ECRL's two-pronged intervention strategy (back-door for background, front-door for data) offers a structured way to address different types of spurious correlations in video data.
  • Event-Level Representation is Key: The success in classifying "fixing bicycle" and similar events demonstrates that focusing on the causal chain of actions is more effective than simply aggregating frame-level object features.

Key Results

  • ECRL achieves superior performance compared to 12 baseline video classification methods on both MSR-VTT and ActivityNet datasets.
  • The model shows significant improvements, particularly on the MSR-VTT dataset, which has a diverse range of events, suggesting ECRL's strength in capturing nuanced event information.
  • Ablation studies confirm the importance of both foreground and background information, as well as the effectiveness of the temporal modeling (TSC), background debiasing (BD), and global-local causal attention (GLCA) modules.
  • Qualitative results (visualizations) demonstrate ECRL's ability to mitigate erroneous predictions caused by background biases and to focus on causally relevant visual information. For instance, it correctly distinguishes "river tubing" from background elements like sky and water, which other models might misinterpret.

Limitations and Future Work

  • Over-intervention: The paper acknowledges that in cases of ambiguous subjects and static backgrounds, ECRL can still make errors due to over-intervention.
  • Future Work: The authors plan to investigate long-term contextual relationships in complex spatio-temporal environments and explore how to better leverage informative multi-modal knowledge within video sequences.

Summary updated

Abstract

Classifying videos differs from that of images in the need to capture the information on what has happened, instead of what is in the frames. Conventional methods typically follow the data-driven approach, which uses transformer-based attention models to extract and aggregate the features of video frames as the representation of the entire video. However, this approach tends to extract the object information of frames and may face difficulties in classifying the classes talking about events, such as "fixing bicycle". To address this issue, This paper presents an Event-level Causal Representation Learning (ECRL) model for the spatio-temporal modeling of both the in-frame object interactions and their cross-frame temporal correlations. Specifically, ECRL first employs a Frame-to-Video Causal Modeling (F2VCM) module, which simultaneously builds the in-frame causal graph with the background and foreground information and models their cross-frame correlations to construct a video-level causal graph. Subsequently, a Causality-aware Event-level Representation Inference (CERI) module is introduced to eliminate the spurious correlations in contexts and objects via the back- and front-door interventions, respectively. The former involves visual context de-biasing to filter out background confounders, while the latter employs global-local causal attention to capture event-level visual information. Experimental results on two benchmarking datasets verified that ECRL may better capture the cross-frame correlations to describe videos in event-level features. The source code is provided in the supplementary material.
BibTeX
@inproceedings{wang2024modeling,
  author = {Yuqing Wang and Lei Meng and Haokai Ma and Yuqing Wang and Haibei Huang and Xiangxu Meng},
  title = {{Modeling Event-level Causal Representation for Video Classification}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681547},
}

Loading related content...

Figure