UCF-STAR: A Large Scale Still Image Dataset for Understanding Human Actions

Marjaneh Safaei, Pooyan Balouchian, Hassan ForooshAAAI 2020 · DOI

Introduction

Recognizing human actions in still images is a fundamental challenge in computer vision. Unlike video-based recognition, where temporal flow provides explicit cues about movement, still images require models to infer intent and motion from a single frozen frame. Historically, this field has been hindered by two primary obstacles: data scarcity and "static bias." Most existing datasets are either too small to train modern deep architectures or rely heavily on object-interaction cues (e.g., identifying "phoning" by the presence of a phone) rather than the nuances of human body motion.

To address these limitations, the UCF-STAR dataset provides a massive-scale resource designed to push the boundaries of action understanding by focusing on the human form and its latent motion.

The UCF-STAR Dataset

UCF-STAR is a comprehensive dataset containing 1,038,622 images across 50 action categories. This represents a nearly 40-fold increase in scale compared to previous benchmarks like BU-101. The dataset was constructed using the Bing Cognitive Services API, employing "semantic grounding" via WordNet to ensure a diverse and accurate collection of motion-centric images.

Overview of the UCF-STAR dataset, illustrating (a) diverse action categories and (b) detailed metadata annotations for individual images.

Beyond its sheer volume, UCF-STAR is distinguished by its rich metadata. Each image is annotated with bounding boxes, descriptive captions, relevant tags, and attributes such as the number of humans present and whether the action involves object interaction. This structured data allows for more sophisticated multi-modal learning and provides a rigorous testing ground for body-motion analysis.

Two-Stream Spatiotemporal Network (TSSTN)

Alongside the dataset, the authors introduced the Two-Stream Spatiotemporal Network (TSSTN). The core innovation of TSSTN is its ability to "hallucinate" missing temporal information from a static image. The architecture consists of two parallel paths:

  • Spatial Stream: A standard Convolutional Neural Network (CNN) that processes the RGB image to identify objects, backgrounds, and static poses.
  • Temporal Stream: This stream utilizes an encoder-decoder model pre-trained on video data. It generates a dynamic-skeleton map—a 2D representation of predicted skeletal movement over a short time window—from the single input frame.

By fusing these streams using Multi-response Regression (MRR), the model can distinguish between visually similar poses that represent different stages of an action, such as the difference between a person simply holding a racket and a person in the middle of a high-velocity tennis swing.

Benchmarking and Results

The TSSTN model was evaluated against several classic benchmarks, consistently achieving state-of-the-art (SOTA) results:

  • UCF-STAR: 91.9% mean Average Precision (mAP).
  • Stanford-40: 86.3% mAP (surpassing the previous SOTA of ~83%).
  • WIDER: 90.3% mAP specifically on body-motion categories.

Ablation studies confirmed that the "hallucinated" temporal stream is a critical component, often outperforming the spatial stream in motion-heavy categories. This proves that human poses in still images contain significant structural information about immediate past and future movements.

Conclusion and Future Work

UCF-STAR sets a new standard for still-image action recognition by providing the scale necessary for deep learning while shifting the focus back to human motion. While the current TSSTN model demonstrates the power of hallucinated motion, future research may further leverage the dataset's rich textual metadata to create even more robust multi-modal architectures.

Summary updated

BibTeX
@inproceedings{safaei2020ucf,
  author = {Marjaneh Safaei and Pooyan Balouchian and Hassan Foroosh},
  title = {{UCF-STAR: A Large Scale Still Image Dataset for Understanding Human Actions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5653},
}
Figure