Leveraging Temporal Context in Low Representational Power Regimes

Camilo Luciano Fosco, SouYoung Jin, Emilie Josephs, Aude OlivaCVPR 2023 · DOI

Abstract

Researchers from MIT CSAIL have introduced the Event Transition Matrix (ETM), a method designed to enhance the performance of small-scale computer vision models in video understanding tasks. By providing "low-power" models with a pre-computed representation of action sequences during training, the authors achieved significant accuracy gains in action recognition and anticipation without increasing computational costs at inference time.

Introduction: The Challenge of Low-Power Video Understanding

State-of-the-art video understanding models typically require substantial computational resources, making them difficult to deploy on edge devices like smartphones or wearables. While large models possess the representational power to learn complex statistical regularities—such as the typical sequence of steps in a task—from scratch, smaller models often lack the capacity to capture these long-range temporal dependencies. The ETM addresses this by explicitly teaching small models about temporal context during the training phase.

Methodology: The Event Transition Matrix (ETM)

The core of this approach is the Event Transition Matrix, a square matrix where each cell represents the likelihood of one action category preceding or following another. Unlike simple Markov chains, the ETM utilizes a decay function to capture long-range relationships, allowing the model to learn that "taking out a pan" is related to "serving food" even if several intermediate steps occur.

Training Protocol

During training, the model's encoder is connected to three distinct prediction heads: Present, Past, and Future. While the Present head performs standard action recognition, the Past and Future heads are trained to predict the distribution of actions likely to have occurred before or after the current snippet, using the ETM as ground truth.

The ETM training protocol: the encoder is trained to predict present, past, and future action distributions using the ETM to provide ground truth for temporal context.

Crucially, the Past and Future heads are discarded after training. This allows the model to benefit from richer internal representations of temporal context during inference without adding any additional parameters or computational overhead during deployment.

Results: The "Small Model" Advantage

The most significant finding of the study is that the performance benefits of ETM are inversely proportional to model size. In experiments on the EPIC-KITCHENS-100 dataset, the tiny MoViNet A0 saw an absolute jump of +3.4% (a 9.24% relative improvement) in action recognition accuracy. In contrast, massive models like AVT-b showed negligible gains (+0.33%).

Performance gain from ETM training as a function of model complexity (GFLOPs), showing that smaller models benefit significantly more than larger ones.

Key Insights

  • Coherence is Essential: "Shuffle" experiments demonstrated that the model specifically requires coherent temporal structure; when the ETM was randomly shuffled, performance gains vanished.
  • Model Agnostic: The ETM approach proved effective across various architectures, including MoViNet, X3D, ConvNeXt, and EfficientNet families.
  • Action Anticipation: The benefits extended to predicting future actions. On the EGO4D Long-Term Anticipation (LTA) task, ETM-trained models improved prediction accuracy by over 1% absolute.

Conclusion and Future Work

The Event Transition Matrix provides a highly efficient way to inject domain-specific temporal knowledge into lightweight models. While currently reliant on dense action labels and global dataset averages, future work may explore self-supervised ETM construction or personalized matrices that adapt to individual user behaviors. This research paves the way for more capable video understanding on resource-constrained edge devices.

Summary updated

BibTeX
@inproceedings{fosco2023leveraging,
  author = {Camilo Luciano Fosco and SouYoung Jin and Emilie Josephs and Aude Oliva},
  title = {{Leveraging Temporal Context in Low Representational Power Regimes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01030},
}
Figure