Title Learning Latent Subevents in Activity Videos Using Temporal Attention Filters

A. J. Piergiovanni, Chenyou Fan, Michael S. Ryoo

Abstract

This paper introduces temporal attention filters, a fully differentiable mechanism that allows deep learning models to automatically discover and focus on the "sub-events" that make up a complex human activity. By replacing simple global pooling with these learnable filters, the researchers achieved state-of-the-art results on challenging activity recognition datasets, effectively teaching the model when and for how long to look at specific parts of a video.

Introduction

Most human activities are not monolithic; they are composed of distinct temporal parts. A "pushup," for example, consists of going down, holding, and coming back up. However, standard deep learning approaches for video recognition often treat videos as a "bag of frames." They typically apply a Convolutional Neural Network (CNN) to individual frames and then use simple global max or average pooling to aggregate the results.

This approach has a major flaw: it completely ignores the temporal structure and the relative duration of sub-events. While some researchers tried using fixed "temporal pyramids" (dividing the video into halves, quarters, etc.), these intervals were predetermined and not optimized for the specific activity. The challenge addressed here is how to make a model explicitly learn this latent temporal structure in an end-to-end, differentiable way without requiring manual labels for every sub-event.

Proposed Architecture

The authors propose a novel layer called a Temporal Attention Filter. These filters sit on top of standard frame-based features (like VGG or C3D) and feed into fully connected layers for final classification. Unlike standard pooling, which collapses the temporal dimension uniformly, these filters learn to attend to specific segments of the video sequence.

Overview of the Temporal Attention Filter architecture, showing the pipeline from raw video input to final classification via learnable filters.

The Filter Mechanism

Each filter is defined by three learnable parameters: a center ($g$), a stride ($\delta$), and a width ($\sigma$). These parameters control a bank of $N$ Gaussian filters that "pick up" features from specific intervals of the video. Because the Gaussian functions are differentiable, the model can use standard backpropagation to update the filter parameters. This means the model learns the optimal location and scale of sub-events directly from the classification error.

Parameters of the Gaussian filter bank: center (g), width (σ), and stride (δ), which define the temporal focus of the attention mechanism.

Static vs. Dynamic Filters

The researchers explored two ways to implement these filters:

  • Static: The model learns a set of filters that are fixed after training and shared across all videos.
  • Dynamic (LSTM-based): Recognizing that people perform actions at different speeds, the authors used a Long Short-Term Memory (LSTM) network to adaptively adjust the filter parameters for each specific video during inference. The LSTM "glimpses" the video and iteratively refines where the filters should focus.

Experimental Results

The researchers evaluated their approach on two prominent benchmarks:

  • DogCentric Activity Dataset: A particularly difficult dataset consisting of first-person videos taken from a camera mounted on a dog. The model achieved 81.4% accuracy, a significant jump from the previous SOTA of 76.6% (TDD).
  • HMDB (Human Motion Database): A large-scale dataset with over 5,000 videos across 51 activity classes. Using TDD features as a base, the temporal filters reached 68.4% accuracy, outperforming other complex architectures like LTC (64.8%) and S+T (65.4%).

Across all feature types (VGG, C3D, ITF, TDD), adding learned temporal filters consistently improved performance compared to max or mean pooling, usually by a margin of 4–10%.

Key Insights

  • Sub-events are Latent Variables: You don't need ground-truth labels for "the moment the arm extends" to learn it. By optimizing for the final action label, the differentiable filters naturally gravitate toward the most discriminative sub-intervals.
  • Resolution Matters: Unlike fixed pyramids, these filters can learn different "temporal resolutions." One filter might focus on a very brief, sharp movement, while another spans a long, slow transition.
  • Semantic Alignment: Qualitative visualizations show that the learned filters actually align with human-understandable sub-events, such as the "roll" and "stand up" phases of a somersault.

Conclusion

While the current model assumes sub-events can be captured by Gaussian-shaped temporal windows, it represents a significant step toward end-to-end temporal modeling. Future work might explore more complex or non-symmetric attention shapes to capture even more nuanced temporal dynamics. Despite the computational overhead of the dynamic LSTM-based approach, the gains in accuracy on ego-centric and real-world datasets demonstrate the power of learning latent temporal structures.

Summary updated

Abstract

In this paper, we newly introduce the concept of temporal attention filters, and describe how they can be used for human activity recognition from videos. Many high-level activities are often composed of multiple temporal parts (e.g., sub-events) with different duration/speed, and our objective is to make the model explicitly learn such temporal structure using multiple attention filters and benefit from them. Our temporal filters are designed to be fully differentiable, allowing end-of-end training of the temporal filters together with the underlying frame-based or segment-based convolutional neural network architectures. This paper presents an approach of learning a set of optimal static temporal attention filters to be shared across different videos, and extends this approach to dynamically adjust attention filters per testing video using recurrent long short-term memory networks (LSTMs). This allows our temporal attention filters to learn latent sub-events specific to each activity. We experimentally confirm that the proposed concept of temporal attention filters benefits the activity recognition, and we visualize the learned latent sub-events.
BibTeX
@inproceedings{piergiovanni2017title,
  author = {A. J. Piergiovanni and Chenyou Fan and Michael S. Ryoo},
  title = {{Title Learning Latent Subevents in Activity Videos Using Temporal Attention Filters}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.1609/aaai.v31i1.11240},
}
Figure