Much Ado About Time: Exhaustive Annotation of Temporal Data

Gunnar A. Sigurdsson, Olga Russakovsky, Ali Farhadi, Ivan Laptev, Abhinav Gupta

Introduction

Annotating video content is a notoriously slow and expensive process, primarily because human annotators must watch footage in real-time to identify relevant actions or objects. While the computer vision community has benefited from massive, exhaustively labeled image datasets like ImageNet, video remains a significant bottleneck. This paper explores the "exhaustive annotation" problem: how to identify every relevant concept in a video efficiently. The authors demonstrate that the most cost-effective approach involves asking crowd workers a high volume of questions (up to 52) after a single viewing, leveraging amortized watching time to outperform traditional methods.

The Challenge of Video Annotation

The fundamental hurdle in video annotation is perception time. While a human can categorize a static image in approximately 150 milliseconds, a 30-second video requires at least 30 seconds of attention (or 15 seconds at double speed). Naive workflows that apply image-tagging logic to video—asking one question per viewing—are prohibitively expensive. If three separate labels are required, the video is typically watched three times by different workers.

Video data introduces a temporal dimension that makes exhaustive annotation significantly more time-consuming than static images.

The Multiple Question Strategy

To address this, the authors propose a Multiple Question Strategy. Instead of the standard one-question-per-video approach, they present workers with dozens of potential labels simultaneously.

Key components of this method include:

  • Semantic Hierarchy: To manage cognitive load, 157 activity labels were grouped into 33 object-based categories (e.g., "interacting with a book"), which then expand into specific actions (e.g., "reading").
  • Amortized Watching Time: By asking many questions after one viewing, the "fixed cost" of watching the video is spread across numerous data points.
  • Consensus through Iteration: While individual recall may drop as the number of questions increases, the speed of the method allows for multiple "cheap" passes. Combining results from several workers yields high-quality aggregate data.
  • Optimized Playback: Utilizing 2x playback speed further reduces the fixed cost without significantly sacrificing worker comprehension.
The cost model for annotation: perception of the visual scene is fast, while answering specific questions adds a small, linear time cost.

Experimental Setup and Datasets

The study utilized the Charades dataset, containing 1,815 videos of daily life activities with 157 human activity classes. Using their multi-question method, the authors collected over 1.3 million annotations. This process increased the label density of the dataset from an average of 3.7 labels per video to 9.0, providing a much richer ground truth for training models.

Results and Discussion

The "many-question" approach was compared against a standard "one-question-per-video" baseline, yielding several key insights:

  • Efficiency and Recall: The strategy achieved a 10% improvement in recall (76.7% vs. 66.7%) while requiring only half the total annotation time (3.8 minutes per video vs. 7.1 minutes).
  • Economic Reality vs. Miller’s Law: Although psychology suggests humans struggle to hold more than seven items in short-term memory, the economic benefits of avoiding re-watching outweigh the drop in individual worker recall.
  • The "Nervous Worker" Effect: Interestingly, providing more options can improve precision. Workers asked only one question may feel pressured to find something and report false positives. A large checklist allows them to accurately select only the few present actions, leading to higher precision (83.8% vs. 83.0%).
  • Scaling with Length: The benefits of this strategy are most pronounced for longer clips (40-60 seconds), where the "watch time" is the dominant cost factor.

Conclusion

This research demonstrates that for video annotation, quantity can lead to quality. By overwhelming workers with questions but reducing the number of times they must watch a video, researchers can collect exhaustive labels faster and more accurately than traditional methods. Future work may explore the upper limits of this cognitive load and how to adapt the strategy for complex temporal tasks like precise timestamping.

Summary updated

Abstract

Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image annotations. These techniques often implicitly rely on the fact that a new input image takes a negligible amount of time to perceive. In contrast, we investigate and determine the most cost-effective way of obtaining high-quality multi-label annotations for temporal data such as videos. Watching even a short 30-second video clip requires a significant time investment from a crowd worker; thus, requesting multiple annotations following a single viewing is an important cost-saving strategy. But how many questions should we ask per video? We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments). We demonstrate that while workers may not correctly answer all questions, the cost-benefit analysis nevertheless favors consensus from multiple such cheap-yet-imperfect iterations over more complex alternatives. When compared with a one-question-per-video baseline, our method is able to achieve a 10% improvement in recall 76.7% ours versus 66.7% baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline). We demonstrate the effectiveness of our method by collecting multi-label annotations of 157 human activities on 1,815 videos.
BibTeX
@inproceedings{sigurdsson2016much,
  author = {Gunnar A. Sigurdsson and Olga Russakovsky and Ali Farhadi and Ivan Laptev and Abhinav Gupta},
  title = {{Much Ado About Time: Exhaustive Annotation of Temporal Data}},
  booktitle = {HCOMP},
  year = {2016},
  doi = {10.1609/hcomp.v4i1.13290},
}
Figure