Abstract
We address the problem of fine-grained action localization from temporally untrimmed web videos. We assume that only weak video-level annotations are available for training. The goal is to use these weak labels to identify temporal segments corresponding to the actions, and learn models that generalize to unconstrained web videos. We find that web images queried by action names serve as well-localized highlights for many actions, but are noisily labeled. To solve this problem, we propose a simple yet effective method that takes weak video labels and noisy image labels as input, and generates localized action frames as output. This is achieved by cross-domain transfer between video frames and web images, using pre-trained deep convolutional neural networks. We then use the localized action frames to train action recognition models with long short-term memory networks. We collect a fine-grained sports action data set FGA-240 of more than 130,000 YouTube videos. It has 240 fine-grained actions under 85 sports activities. Convincing results are shown on the FGA-240 data set, as well as the THUMOS 2014 localization data set with untrimmed training videos.
Introduction
The challenge of action recognition in the wild lies in the nature of web videos, which are typically "untrimmed." Unlike curated datasets where clips are precisely cut to the action, a ten-minute YouTube video might contain only a few seconds of the target activity, such as a slam dunk. Manually annotating these temporal boundaries is prohibitively expensive. This research addresses the problem of fine-grained action localization using only weak video-level labels, leveraging the abundance of web images as a bridge to identify relevant frames.
Methodology: Cross-Domain Transfer
The proposed framework utilizes a pre-trained Convolutional Neural Network (CNN) to map both video frames and web images into a shared visual space. The core innovation is a bidirectional transfer process:
- Video to Image: Video frames are used to train a classifier that filters out irrelevant noise from web image search results (e.g., logos or advertisements).
- Image to Video: These filtered, high-quality images—which naturally capture the "peak" of an action—are used to train a classifier that assigns a Localized Action Frame (LAF) score to each video frame.
These LAF scores then act as weights for training a Long Short-Term Memory (LSTM) network. By down-weighting the loss for frames with low LAF scores, the model is forced to focus its learning on the most semantically relevant segments of the untrimmed video.
Datasets and Experimental Setup
The study introduces FGA-240, a large-scale dataset comprising over 130,000 YouTube videos across 240 fine-grained categories (e.g., distinguishing a jump shot from a layup). The method was also validated on the THUMOS 2014 benchmark to demonstrate its generalizability in temporal localization tasks.
Key Insights and Results
A primary insight is that web images act as naturally occurring "highlights." While search results can be noisy, they are almost always "trimmed" to the most descriptive moment of an action. Furthermore, the "noise" in the two domains is complementary; video backgrounds (stadiums) differ significantly from image search noise (product shots), making it easier for the model to isolate the shared action signal.
Empirical results on FGA-240 show that the LAF-weighted LSTM achieves a 43.4% Hit@1, significantly outperforming standard CNN and unweighted LSTM baselines. On THUMOS 2014, the method demonstrated superior localization performance, achieving 12.4% mAP at a 0.1 overlap threshold, compared to 9.8% for the baseline.
Conclusion and Future Work
While effective, the domain transfer approach faces challenges when visual styles or viewpoints diverge significantly between images and videos (e.g., professional photography vs. GoPro footage). Future research may explore incorporating motion features like optical flow or developing more robust methods to handle viewpoint mismatches.
Summary updated