Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks

Matthew Kowal, Mennatullah Siam, Md. Amirul Islam, Neil D. B. Bruce, Richard P. Wildes, Konstantinos G. Derpanis

TL;DR

This paper introduces a novel method to quantify the "static" (appearance-based) versus "dynamic" (motion-based) information captured by deep video understanding models. They discover most models are surprisingly biased towards static information, even on datasets thought to be dynamic, and propose techniques like StaticDropout to encourage learning dynamic features, improving performance on relevant tasks.

What Problem Does This Paper Address?

Deep neural networks for video understanding, like action recognition and video segmentation, have achieved impressive results. However, there's a significant lack of understanding about what information these models actually learn in their intermediate representations. Specifically, it's unclear whether they rely more on static visual cues (like object shapes and textures from a single frame) or dynamic cues (like motion and temporal changes across frames). This "black box" nature is problematic for scientific understanding and for deploying these models responsibly. Existing methods for analyzing this are often qualitative or lack a quantitative way to compare static versus dynamic biases.

Methods

The core of this work is a new methodology for quantifying static and dynamic biases.

  1. Static and Dynamic Pair Sampling: They create pairs of videos where either the static information is preserved but dynamics are altered (static pairs), or vice-versa (dynamic pairs). This is achieved through techniques like video stylization (to change appearance) and frame shuffling or optical flow jitter (to alter temporal dynamics).
  2. Mutual Information Estimation: Using a pre-trained spatiotemporal model, they feed these video pairs through the network and compute the mutual information between the intermediate representations of each pair. This allows them to estimate how much a specific layer or even an individual "unit" (channel) in the network is sensitive to static versus dynamic information.
  3. Unit-wise Bias Analysis: They further categorize units as primarily static, dynamic, jointly encoding both, or residual (uncorrelated with either).
  4. Model Biasing Techniques: Based on their findings, they propose:
  5. StaticDropout: A dropout technique for action recognition that specifically targets and drops out static-biased units during training, encouraging the model to learn more dynamic features.
  6. Fusion and Cross-Connection Design: For AVOS, they analyze how different fusion and cross-connection strategies in two-stream networks influence static/dynamic biases and propose better configurations.

Datasets

The study utilizes a variety of datasets for training and evaluation across different tasks:

  • Action Recognition: Kinetics-400, Diving48, Something-Something-v2 (SSv2), ActivityNet, and SomethingElse.
  • Video Object Segmentation (AVOS): DAVIS16, Weakly Labelled ImageNet VID, YouTube-VOS, and TAO-VOS.
  • Video Instance Segmentation (VIS): YouTube-VIS 2019 (YTVIS) and Occluded Video Instance Segmentation (OVIS).

They note that SSv2 is better suited for evaluating dynamic understanding than Diving48, contrary to prior assumptions.

Key Insights and Ideas

  • Pervasive Static Bias: Most examined models across all tasks are heavily biased towards static information, even on datasets assumed to be dynamic. This is a significant finding that challenges common assumptions.
  • Dataset Nuances: Datasets thought to be dynamic (like Diving48) can actually be biased towards static information, while others (like SSv2) are better for evaluating dynamic learning.
  • Channel-Level Specialization: Individual channels within a network can specialize in encoding static, dynamic, or a combination of both, offering a fine-grained view of representation.
  • Early Learning of Biases: Models tend to converge to their final static and dynamic biases within the first half of training, suggesting that early training phases are crucial for establishing these preferences.
  • Two-Stream Architectures Help: Models with two-stream architectures (e.g., SlowFast) and specific cross-connections (motion-to-appearance) show a greater ability to capture dynamic information.
  • Tunable Biases: The proposed StaticDropout technique demonstrates that static biases can be actively controlled, leading to improved performance on dynamic-centric tasks.

Key Results

  • Action Recognition:
    • Most 3D CNNs and transformer models show a strong static bias.
  • SlowFast's fast branch exhibits a notable number of dynamic units.
  • StaticDropout successfully debiased models, increasing dynamic units and improving performance on SSv2, especially for rare classes. For example, on SSv2, StaticDropout with r=0.5 increased dynamic units to 45.1% and improved Top-1 accuracy.
  • AVOS:
    • Fusion layers in AVOS models are generally biased towards static information.
  • MATNet, with its motion-to-appearance cross-connections, showed a better balance of dynamic and static units compared to FusionSeg and RTNet.
  • Training on TAO-VOS resulted in more specialized dynamic units compared to other AVOS datasets.
  • VIS:
    • VisTR models are strongly biased towards static information across all layers.
  • The 2D backbone architecture plays a role in the static/dynamic unit ratio.
  • Training Dynamics: Static and dynamic information are largely learned in the first half of training for most models and datasets.

Limitations and Future Work

  • Confounder of Shape: The paper acknowledges that shape is an inherent confounder with static and dynamic information, as perceiving motion often implies perceiving object boundaries.
  • Dataset Bias vs. Task Bias: While they analyze dataset biases, the ultimate goal is to understand how these affect performance on tasks that require dynamics.
  • Optimal Balance: For action recognition, maximizing dynamic units with StaticDropout wasn't always optimal; a balance between static and dynamic information is likely needed.
  • Future Directions: The authors suggest exploring more sophisticated self-attention mechanisms for VIS to better admit specialized dynamic units, and applying their methodology to other video understanding tasks like action prediction. They also propose further investigation into the interaction between models and datasets for more stable bias learning.

Summary updated

Abstract

There is limited understanding of the information captured by deep spatiotemporal models in their intermediate representations. For example, while evidence suggests that action recognition algorithms are heavily influenced by visual appearance in single frames, no quantitative methodology exists for evaluating such static bias in the latent representation compared to bias toward dynamics. We tackle this challenge by proposing an approach for quantifying the static and dynamic biases of any spatiotemporal model, and apply our approach to three tasks, action recognition, automatic video object segmentation (AVOS) and video instance segmentation (VIS). Our key findings are: (i) Most examined models are biased toward static information. (ii) Some datasets that are assumed to be biased toward dynamics are actually biased toward static information. (iii) Individual channels in an architecture can be biased toward static, dynamic or a combination of the two. (iv) Most models converge to their culminating biases in the first half of training. We then explore how these biases affect performance on dynamically biased datasets. For action recognition, we propose StaticDropout, a semantically guided dropout that debiases a model from static information toward dynamics. For AVOS, we design a better combination of fusion and cross connection layers compared with previous architectures.
BibTeX
@article{kowal2022quantifying,
  author = {Matthew Kowal and Mennatullah Siam and Md. Amirul Islam and Neil D. B. Bruce and Richard P. Wildes and Konstantinos G. Derpanis},
  title = {{Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=G85_fTV7mA},
}
Figure