Fine-grained Activities of People Worldwide

Jeffrey Byrne, Greg Castañón, Zhongheng Li, Gil J. EttingerWACV 2023 · DOI

Abstract

The Consented Activities of People (CAP) dataset is a massive new resource for video AI, featuring 1.45 million clips of 512 fine-grained daily activities performed by 780 subjects across 33 countries. Unlike traditional datasets scraped from social media, CAP was built ethically using a custom mobile platform ("Collector") to ensure subject consent and high-quality, diverse annotations for subtle motions. This paper details the collection methodology, dataset characteristics, and benchmarks demonstrating significant performance improvements in activity detection tasks.

Introduction

Most large-scale activity recognition datasets, such as Kinetics or YouTube-8M, suffer from three major flaws: ethical concerns regarding non-consensual web scraping, a lack of granularity for subtle daily motions (e.g., "opening a jar" vs. "closing a jar"), and significant geographic bias toward Western cultures. For AI to be useful in real-world applications—like assistive robotics or ethical security monitoring—it needs to understand these subtle, fundamental human actions across a global context.

Methods: The Collector Platform

The researchers developed a novel collection platform called "Collector." Instead of scraping the web, they built a mobile app that allowed a global workforce of collectors to record videos of consented subjects. The app includes a touchscreen UI for live annotation of bounding boxes and activity timestamps, and it features in-app face anonymization to protect privacy.

The Collector mobile application workflow, showing the process from project selection and subject consent to recording and peer rating.

To define "fine-grained" activities, the team used a "Noun-Verb" structure, focusing on activities that are closely related and easily confused (e.g., "person puts on socks" vs. "person puts on shoes"). This ensures the AI must learn subtle discriminative motions rather than relying on static scene context.

The CAP Dataset

The primary contribution is the Consented Activities of People (CAP) dataset. With 1.45 million video clips, it stands out as the largest consented dataset by mean clips per class. It features 512 fine-grained labels and 144 coarse labels, collected across 104 cities in 33 countries, providing a more diverse geographic representation than typical social media datasets.

Overview of the CAP dataset, highlighting its scale (1.45M clips), global distribution of collectors, and examples of fine-grained activity detection.

Key Insights and Results

Benchmarking CAP against existing datasets reveals several critical insights:

  • Performance Boost: Training on CAP data and testing on the ActEV SDL security leaderboard showed a 32% improvement in detection performance compared to training on the MEVA dataset alone.
  • Causal Pair Confusion: The researchers found that "causal pairs"—actions that naturally follow one another, like "open" and "close"—are the most commonly confused by AI because they often co-occur in short sequences.
  • Scene Context Bias: While models perform well on classes with heavy scene context (like "loads dishwasher"), they struggle with subtle movements (like "puts up smoke detector"), suggesting that current models still rely heavily on background cues rather than human motion.
  • Scalability: The "Collector" model proves that ethical data collection is scalable and can produce higher-quality, more diverse data than web scraping.

Limitations and Future Work

Despite its scale, the dataset faces challenges with label imbalance, as "organic" activities like "sitting down" occur much more frequently than specific tasks. Additionally, while classification is improving, precise temporal detection (exactly when an action starts and stops) remains a significant challenge. Future work will focus on expanding the Collector platform into specialized domains like healthcare and industrial safety.

Summary updated

BibTeX
@inproceedings{byrne2023fine,
  author = {Jeffrey Byrne and Greg Castañón and Zhongheng Li and Gil J. Ettinger},
  title = {{Fine-grained Activities of People Worldwide}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00332},
}
Figure