TL;DR
This paper introduces a novel method to quantify the "static" (appearance-based) versus "dynamic" (motion-based) information captured by deep video understanding models. They discover most models are surprisingly biased towards static information, even on datasets thought to be dynamic, and propose techniques like StaticDropout to encourage learning dynamic features, improving performance on relevant tasks.
What Problem Does This Paper Address?
Deep neural networks for video understanding, like action recognition and video segmentation, have achieved impressive results. However, there's a significant lack of understanding about what information these models actually learn in their intermediate representations. Specifically, it's unclear whether they rely more on static visual cues (like object shapes and textures from a single frame) or dynamic cues (like motion and temporal changes across frames). This "black box" nature is problematic for scientific understanding and for deploying these models responsibly. Existing methods for analyzing this are often qualitative or lack a quantitative way to compare static versus dynamic biases.
Methods
The core of this work is a new methodology for quantifying static and dynamic biases.
- Static and Dynamic Pair Sampling: They create pairs of videos where either the static information is preserved but dynamics are altered (static pairs), or vice-versa (dynamic pairs). This is achieved through techniques like video stylization (to change appearance) and frame shuffling or optical flow jitter (to alter temporal dynamics).
- Mutual Information Estimation: Using a pre-trained spatiotemporal model, they feed these video pairs through the network and compute the mutual information between the intermediate representations of each pair. This allows them to estimate how much a specific layer or even an individual "unit" (channel) in the network is sensitive to static versus dynamic information.
- Unit-wise Bias Analysis: They further categorize units as primarily static, dynamic, jointly encoding both, or residual (uncorrelated with either).
- Model Biasing Techniques: Based on their findings, they propose:
- StaticDropout: A dropout technique for action recognition that specifically targets and drops out static-biased units during training, encouraging the model to learn more dynamic features.
- Fusion and Cross-Connection Design: For AVOS, they analyze how different fusion and cross-connection strategies in two-stream networks influence static/dynamic biases and propose better configurations.
Datasets
The study utilizes a variety of datasets for training and evaluation across different tasks:
- Action Recognition: Kinetics-400, Diving48, Something-Something-v2 (SSv2), ActivityNet, and SomethingElse.
- Video Object Segmentation (AVOS): DAVIS16, Weakly Labelled ImageNet VID, YouTube-VOS, and TAO-VOS.
- Video Instance Segmentation (VIS): YouTube-VIS 2019 (YTVIS) and Occluded Video Instance Segmentation (OVIS).
They note that SSv2 is better suited for evaluating dynamic understanding than Diving48, contrary to prior assumptions.
Key Insights and Ideas
- Pervasive Static Bias: Most examined models across all tasks are heavily biased towards static information, even on datasets assumed to be dynamic. This is a significant finding that challenges common assumptions.
- Dataset Nuances: Datasets thought to be dynamic (like Diving48) can actually be biased towards static information, while others (like SSv2) are better for evaluating dynamic learning.
- Channel-Level Specialization: Individual channels within a network can specialize in encoding static, dynamic, or a combination of both, offering a fine-grained view of representation.
- Early Learning of Biases: Models tend to converge to their final static and dynamic biases within the first half of training, suggesting that early training phases are crucial for establishing these preferences.
- Two-Stream Architectures Help: Models with two-stream architectures (e.g., SlowFast) and specific cross-connections (motion-to-appearance) show a greater ability to capture dynamic information.
- Tunable Biases: The proposed StaticDropout technique demonstrates that static biases can be actively controlled, leading to improved performance on dynamic-centric tasks.
Key Results
- Action Recognition:
- Most 3D CNNs and transformer models show a strong static bias.
- SlowFast's fast branch exhibits a notable number of dynamic units.
- StaticDropout successfully debiased models, increasing dynamic units and improving performance on SSv2, especially for rare classes. For example, on SSv2, StaticDropout with r=0.5 increased dynamic units to 45.1% and improved Top-1 accuracy.
- AVOS:
- Fusion layers in AVOS models are generally biased towards static information.
- MATNet, with its motion-to-appearance cross-connections, showed a better balance of dynamic and static units compared to FusionSeg and RTNet.
- Training on TAO-VOS resulted in more specialized dynamic units compared to other AVOS datasets.
- VIS:
- VisTR models are strongly biased towards static information across all layers.
- The 2D backbone architecture plays a role in the static/dynamic unit ratio.
- Training Dynamics: Static and dynamic information are largely learned in the first half of training for most models and datasets.
Limitations and Future Work
- Confounder of Shape: The paper acknowledges that shape is an inherent confounder with static and dynamic information, as perceiving motion often implies perceiving object boundaries.
- Dataset Bias vs. Task Bias: While they analyze dataset biases, the ultimate goal is to understand how these affect performance on tasks that require dynamics.
- Optimal Balance: For action recognition, maximizing dynamic units with StaticDropout wasn't always optimal; a balance between static and dynamic information is likely needed.
- Future Directions: The authors suggest exploring more sophisticated self-attention mechanisms for VIS to better admit specialized dynamic units, and applying their methodology to other video understanding tasks like action prediction. They also propose further investigation into the interaction between models and datasets for more stable bias learning.
Summary updated