RIT-18: A Novel Dataset for Compositional Group Activity Understanding

Junwen Chen, Haiting Hao, Hanbin Hong, Yu KongWorkshops @ CVPR 2020 · OpenReview

Abstract

Group activity understanding is a challenging task as multiple people are involved, and their relations may vary over time. Currently, the literature of group activity is limited to group activity recognition, because videos are trimmed in very short duration and focus on a single activity. This slows down the progress in the group activity domain. In this paper, we propose a new large-scale untrimmed compositional group activity dataset RIT-18 based on the volleyball games captured from YouTube. Each clip in our dataset depicts an entire rally which spans the duration from serve to a point being scored. Comprehensive annotations including group activity labels, temporal boundaries of activities, key persons, and winning teams are provided. We describe group activity recognition, future activity anticipation, and rally-level winner prediction challenges, and evaluate several baseline methods over these challenges. We report their performance on our dataset and demonstrate further efforts need to be made.

Introduction: Moving Beyond Snippets

The RIT-18 dataset shifts the focus of group activity understanding from identifying isolated, short-duration actions to analyzing entire "compositional" sequences. Most research in group activity recognition (GAR) has been limited by trimmed video snippets. Popular benchmarks, such as the original Volleyball Dataset, often provide only about 40 frames centered around a single action, like a spike. While sufficient for basic classification, this does not reflect real-world scenarios where activities are continuous, evolving chains of events.

In applications like autonomous driving or surveillance, it is not enough to know what is happening in the moment; we must anticipate future actions and understand the overarching goals of a group. The lack of long-form, untrimmed data has hindered progress toward these complex, compositional understanding tasks.

Overview of the RIT-18 tasks: group activity recognition, future activity anticipation, and winner prediction across an untrimmed volleyball rally.

The RIT-18 Dataset

The authors introduce RIT-18, a dataset designed to treat group activity as a sequence of related sub-events.

  • Scale and Source: It comprises 1,530 clips sourced from 51 YouTube videos of professional and collegiate volleyball, including the Olympics and NCAA.
  • Composition: Each clip represents a full rally from serve to point scored.
  • Annotations: The dataset includes 18 activity classes (e.g., l-serve, r-spike, l-block), temporal boundaries for every sub-activity, bounding boxes for the "key person" touching the ball, and the winner of each rally.
  • Comparison: Unlike the 2016 Volleyball Dataset, which features 8 classes and 41-frame clips, RIT-18 offers 18 classes and an average clip length of approximately 300 frames (9.09 seconds).
Demographic distribution of the RIT-18 dataset by gender, year, and competition type.

Methods and Challenges

To establish benchmarks, three state-of-the-art models were evaluated:

  1. HDTM (Hierarchical Deep Temporal Model): Designed to capture temporal dynamics.
  2. SSU (Social Scene Understanding): A framework that integrates person detection and activity recognition.
  3. ARG (Actor Relation Graph): A method that builds a learnable graph to model relationships between individuals.

For the novel task of Winner Prediction, three specific baselines were developed:

  • B1 (LSTM): A standard sequence model using VGG19 features.
  • B2 (Extended ARG): Adds temporal ordering to the Actor Relation Graph.
  • B3 (Monotone Importance): An ARG variant that weights frames closer to the rally's conclusion (the "winpoint") more heavily, reflecting their higher indicative value for the final outcome.

Key Insights and Results

  • Compositional Complexity: Tracking the flow of a rally is significantly harder than identifying isolated actions. The mean accuracy of the ARG model dropped from ~92% on the old Volleyball dataset to 68.09% on RIT-18, highlighting the increased difficulty of untrimmed sequences.
  • The "Rare Activity" Hurdle: Models struggle with infrequent activities like "shots" or "drops," which are visually inconsistent. The ARG model achieved 0% accuracy on the l-shot class.
  • Anticipation vs. Recognition: Activity anticipation (predicting the next move from the first 10% of an action) is a major challenge. Accuracy improved from ~37% at 10% observation to ~50% at 70% observation.
  • Winner Prediction: Standard LSTMs performed near chance (51.25%). However, the Extended ARG with Monotone Importance (B3) reached 75.61% accuracy, demonstrating that modeling interpersonal relations and temporal progression is crucial for understanding group intent.

Limitations and Future Work

Current models fail on rare classes, necessitating better handling of class imbalance. Future research could extend to "Game-level" prediction, using accumulated scores and momentum to predict match outcomes. Furthermore, there is significant room to improve how models learn the specific inter-person interactions that signal shifts in game flow.

Summary updated

Abstract

Group activity understanding is a challenging task as multiple people are involved, and their relations may vary over time. Currently, the literature of group activity is limited to group activity recognition, because videos are trimmed in very short duration and focus on a single activity. This slows down the progress in the group activity domain. In this paper, we propose a new large-scale untrimmed compositional group activity dataset RIT-18 based on the volleyball games captured from YouTube. Each clip in our dataset depicts an entire rally which spans the duration from serve to a point being scored. Comprehensive annotations including group activity labels, temporal boundaries of activities, key persons, and winning teams are provided. We describe group activity recognition, future activity anticipation, and rally-level winner prediction challenges, and evaluate several baseline methods over these challenges. We report their performance on our dataset and demonstrate further efforts need to be made. The dataset is available at https://pht180.rit.edu/actionlab/rit-18.
BibTeX
@inproceedings{chen2020rit,
  author = {Junwen Chen and Haiting Hao and Hanbin Hong and Yu Kong},
  title = {{RIT-18: A Novel Dataset for Compositional Group Activity Understanding}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=6EXkb7LDss3},
}
Figure