Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images

Chen Sun, Sanketh Shetty, Rahul Sukthankar, Ram Nevatia

TL;DR

This paper introduces a method to automatically find "fine-grained" actions (like a slam dunk or tennis serve) within long, untrimmed web videos using only video-level labels. By leveraging noisy but well-localized web images as a bridge, the authors develop a "Localised Action Frame" (LAF) proposal system that significantly improves temporal localization and classification.

What Problem Does This Paper Address?

Most action recognition research happens on "trimmed" clips—short videos where the action starts at the beginning and ends at the end. Real-world web videos (like those on YouTube) are "untrimmed": a 10-minute basketball video might contain only five seconds of a slam dunk.

Manually labeling exactly when these actions occur (temporal localization) is incredibly expensive. While "weak labels" (knowing a video contains a dunk somewhere) are easy to get, they don't tell a model which frames actually matter. The challenge is: how do we train a model to find these needles in the haystack without expensive frame-by-frame annotations?

Methods

The authors propose a framework that uses Domain Transfer from web images to supervise video learning.

  1. Shared Representation: They use a pre-trained Convolutional Neural Network (CNN) to extract features from both video frames and web images, placing them in a shared visual space.
  2. Bidirectional Domain Transfer:
    • Video to Image: They use video frames to train a classifier that filters out "non-video-like" noise from web image search results (like logos or advertisements).
  3. Image to Video: The filtered, high-quality web images are then used to train a classifier that assigns a Localized Action Frame (LAF) score to every frame in the untrimmed videos.
  4. Temporal Modeling with LSTM: Finally, they train a Long Short-Term Memory (LSTM) network on the video sequences. Crucially, they use the LAF scores as "weights" for the loss function. If the LAF score is low (meaning the frame likely doesn't contain the action), the model isn't penalized much for misclassifying it. This forces the LSTM to focus on the most relevant segments.

Datasets

  • FGA-240 (Fine-Grained Actions): A new large-scale dataset introduced in this paper. It contains over 130,000 YouTube videos across 240 fine-grained categories (e.g., jump shot vs. layup) under 85 sports activities.
  • THUMOS 2014: A standard benchmark for action localization. The authors specifically tested on the untrimmed validation and test sets to prove the generalizability of their domain transfer approach.

Key Insights and Ideas

  • Web Images as "Highlights": The core "aha moment" is realizing that web image search results for an action (e.g., "baseball pitch") act as naturally occurring "highlights." While noisy, they are almost always "trimmed" to the peak of the action.
  • Complementary Noise: Filtering is easier across domains. The "background" of a basketball video is a stadium, but the "noise" in a Google Image search for basketball is often a product shot of a shoe. Because the irrelevant data looks different in each domain, the model can more easily identify the shared signal: the action itself.
  • Soft Supervision: Instead of a hard "yes/no" on whether a frame contains an action, the LAF score provides a continuous weight. This allows the LSTM to learn temporal context (what happens right before a dunk) without being strictly forced to label those "setup" frames as the action itself.

Key Results

  • Classification Boost: On the FGA-240 dataset, the full "LSTM w/ LAF" model achieved a 43.4% Hit@1, outperforming a standard CNN baseline (37.3%) and an LSTM without the image-transfer scores (41.1%).
  • Localization Superiority: The method showed its real strength in finding actions. On THUMOS 2014, the LAF-weighted LSTM significantly outperformed the "Video-only" baseline across all overlap thresholds. For example, at a 0.1 overlap ratio, it achieved 12.4% mAP compared to the baseline's 9.8%.
  • Hierarchy Matters: The authors found that while the model is great at identifying the general activity (e.g., "Basketball"), the major source of error is "fine-grained" confusion (e.g., mistaking a slam dunk for an alley-oop).

Limitations and Future Work

The authors acknowledge that domain transfer fails when web images are "too different" from video frames. This happens with:

  • Beautified Images: Highly edited photos or those with heavy logos (common in Parkour).
  • Viewpoint Mismatch: Web images of Paragliding are often taken from the ground looking up, while web videos are often GoPro footage from the pilot's perspective.

Future work could involve more sophisticated ways to handle these viewpoint shifts or incorporating motion features (like optical flow) directly into the domain transfer process, rather than relying solely on static visual features.

Summary updated

Abstract

We address the problem of fine-grained action localization from temporally untrimmed web videos. We assume that only weak video-level annotations are available for training. The goal is to use these weak labels to identify temporal segments corresponding to the actions, and learn models that generalize to unconstrained web videos. We find that web images queried by action names serve as well-localized highlights for many actions, but are noisily labeled. To solve this problem, we propose a simple yet effective method that takes weak video labels and noisy image labels as input, and generates localized action frames as output. This is achieved by cross-domain transfer between video frames and web images, using pre-trained deep convolutional neural networks. We then use the localized action frames to train action recognition models with long short-term memory networks. We collect a fine-grained sports action data set FGA-240 of more than 130,000 YouTube videos. It has 240 fine-grained actions under 85 sports activities. Convincing results are shown on the FGA-240 data set, as well as the THUMOS 2014 localization data set with untrimmed training videos.
BibTeX
@inproceedings{sun2015temporal,
  author = {Chen Sun and Sanketh Shetty and Rahul Sukthankar and Ram Nevatia},
  title = {{Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images}},
  booktitle = {ACM Multimedia},
  year = {2015},
  url = {https://openreview.net/forum?id=gs0THaiHzE},
}

Loading related content...

Figure