TL;DR
This paper introduces a novel method for zero-shot compositional video understanding, inspired by how humans learn by combining existing concepts. By employing an information-theoretic approach called "coding rate reduction," the model learns to disentangle video representations into distinct spatial and temporal semantic units, leading to state-of-the-art performance on challenging compositional generalization benchmarks.
What Problem Does This Paper Address?
The paper tackles the challenge of compositional generalization in video understanding. While large-scale visual language models (VLMs) have made impressive strides, they often struggle with tasks requiring the combination of learned concepts into novel scenarios, especially in the context of video. This is particularly true for zero-shot learning, where models must generalize to unseen combinations of actions, objects, and temporal dynamics without explicit training examples. Existing methods often require massive datasets and computational resources, and their generalization capabilities are not fully understood. The authors aim to bridge this gap by developing a method that mimics human learning, where new concepts are formed by flexibly recombining existing knowledge.
Methods
The core of the proposed method lies in a novel information-theoretic approach for video representation learning that focuses on coding rate reduction. This technique, inspired by rate-distortion theory, aims to find representations that are both compressed (requiring fewer bits to encode) and disentangled (meaningful semantic units are separated).
The architecture is based on a resampler-like transformer decoder. This module takes features from a pre-trained video encoder and iteratively refines them using cross-attention and feed-forward networks. Crucially, at each iteration, a coding rate reduction optimization is applied. This process encourages the model to learn representations that are highly compressed within specific semantic subspaces (e.g., spatial or temporal) while being incoherent between different subspaces. This effectively disentangles the spatiotemporal information in videos into multiple hierarchical subspaces.
The method learns M latent queries that are updated iteratively. Finally, these resampled representations are passed through two separate heads (spatial and temporal) to produce final representations, which are then used for action prediction. The training objective involves minimizing cross-entropy losses for composite actions and their primitive spatial and temporal categories.
Datasets
The paper evaluates its method on the following datasets:
- Sth-Com dataset: A large-scale benchmark for evaluating zero-shot compositional action recognition, derived from Something-Something V2. It requires models to predict unseen action classes formed by combinations of known verbs and objects.
- CATER dataset: Another challenging benchmark for compositional action recognition and temporal reasoning, featuring synthetic videos with multiple objects performing actions. A specific setting based on this dataset is used for zero-shot compositional action recognition.
- C-GQA dataset: Used to demonstrate the method's applicability to image domain compositional generalization. This dataset is curated from Stanford GQA and focuses on recognizing novel object-attribute compositions.
Key Insights and Ideas
- Human-like learning through composition: The paper draws inspiration from how children learn by combining existing concepts, suggesting this is a powerful paradigm for achieving robust generalization.
- Coding Rate Reduction for Disentanglement: The core innovation is the application of coding rate reduction as an information-theoretic measure to achieve spatiotemporally disentangled representations. This moves beyond simply learning features to learning structured and interpretable features.
- Hierarchical Subspaces: The method explicitly aims to disentangle video features into multiple hierarchical subspaces, allowing for a more fine-grained understanding of spatial and temporal dynamics.
- Interpretable Representations: The authors highlight that their method learns interpretable representations, which is a significant advantage over many black-box deep learning models.
- Effectiveness beyond Video: The successful application to the C-GQA dataset demonstrates that the core principles of disentangled representation learning are transferable across modalities.
Key Results
- State-of-the-Art Performance: The proposed method achieves state-of-the-art results on the Sth-Com and CATER datasets for zero-shot compositional action recognition.
- Sth-Com: Outperforms previous methods by a significant margin, even without relying on explicit label encodings or complex training strategies. When using a VLM backbone (CLIP), it shows even greater improvement.
- CATER: Achieves the highest score on the unseen split with the closest gap between seen and unseen performance, outperforming strong VLM baselines like InternVL2-8B and Qwen2-VL-8B.
- C-GQA: Demonstrates strong performance, outperforming existing zero-shot compositional learning approaches on this image dataset.
- Qualitative Results: Attention map visualizations show that the model accurately captures informative objects and their movements, even without dense object position labels. PCA analysis of resampled representations reveals that the learned subspaces are closely aligned with semantic axes of attributes.
Limitations and Future Work
- Computational Resources: While the paper aims to be more efficient than some large VLMs, the training process still involves significant computation, especially for the iterative resampler module.
- Hyperparameter Sensitivity: The paper notes that using too many latent queries (M) can decrease performance, suggesting a need for careful hyperparameter tuning. The optimal number of subspaces (K) was found to be 24.
- Future Directions: The authors suggest that their approach offers a new direction for tackling compositional generalization in video understanding. Further exploration of its application to other challenging tasks like visual question answering is implied. The success on C-GQA also opens avenues for extending this to other vision-language tasks.
Summary updated