Introduction
Annotating video content is a notoriously slow and expensive process, primarily because human annotators must watch footage in real-time to identify relevant actions or objects. While the computer vision community has benefited from massive, exhaustively labeled image datasets like ImageNet, video remains a significant bottleneck. This paper explores the "exhaustive annotation" problem: how to identify every relevant concept in a video efficiently. The authors demonstrate that the most cost-effective approach involves asking crowd workers a high volume of questions (up to 52) after a single viewing, leveraging amortized watching time to outperform traditional methods.
The Challenge of Video Annotation
The fundamental hurdle in video annotation is perception time. While a human can categorize a static image in approximately 150 milliseconds, a 30-second video requires at least 30 seconds of attention (or 15 seconds at double speed). Naive workflows that apply image-tagging logic to video—asking one question per viewing—are prohibitively expensive. If three separate labels are required, the video is typically watched three times by different workers.
The Multiple Question Strategy
To address this, the authors propose a Multiple Question Strategy. Instead of the standard one-question-per-video approach, they present workers with dozens of potential labels simultaneously.
Key components of this method include:
- Semantic Hierarchy: To manage cognitive load, 157 activity labels were grouped into 33 object-based categories (e.g., "interacting with a book"), which then expand into specific actions (e.g., "reading").
- Amortized Watching Time: By asking many questions after one viewing, the "fixed cost" of watching the video is spread across numerous data points.
- Consensus through Iteration: While individual recall may drop as the number of questions increases, the speed of the method allows for multiple "cheap" passes. Combining results from several workers yields high-quality aggregate data.
- Optimized Playback: Utilizing 2x playback speed further reduces the fixed cost without significantly sacrificing worker comprehension.
Experimental Setup and Datasets
The study utilized the Charades dataset, containing 1,815 videos of daily life activities with 157 human activity classes. Using their multi-question method, the authors collected over 1.3 million annotations. This process increased the label density of the dataset from an average of 3.7 labels per video to 9.0, providing a much richer ground truth for training models.
Results and Discussion
The "many-question" approach was compared against a standard "one-question-per-video" baseline, yielding several key insights:
- Efficiency and Recall: The strategy achieved a 10% improvement in recall (76.7% vs. 66.7%) while requiring only half the total annotation time (3.8 minutes per video vs. 7.1 minutes).
- Economic Reality vs. Miller’s Law: Although psychology suggests humans struggle to hold more than seven items in short-term memory, the economic benefits of avoiding re-watching outweigh the drop in individual worker recall.
- The "Nervous Worker" Effect: Interestingly, providing more options can improve precision. Workers asked only one question may feel pressured to find something and report false positives. A large checklist allows them to accurately select only the few present actions, leading to higher precision (83.8% vs. 83.0%).
- Scaling with Length: The benefits of this strategy are most pronounced for longer clips (40-60 seconds), where the "watch time" is the dominant cost factor.
Conclusion
This research demonstrates that for video annotation, quantity can lead to quality. By overwhelming workers with questions but reducing the number of times they must watch a video, researchers can collect exhaustive labels faster and more accurately than traditional methods. Future work may explore the upper limits of this cognitive load and how to adapt the strategy for complex temporal tasks like precise timestamping.
Summary updated