EventNet Version 1.1 Technical Report

Dongang Wang, Zheng Shou, Hongyi Liu, Shih-Fu ChangarXiv 2016 · OpenReview

TL;DR

This paper presents EventNet v1.1, a significantly improved version of a large-scale video corpus for event recognition. By meticulously cleaning and augmenting the dataset, they've created a more robust resource that addresses common biases and data quality issues, making it a better foundation for training advanced event classification models.

What Problem Does This Paper Address?

The original EventNet dataset, while large, suffered from several quality issues that hindered its effectiveness for training computer vision and multimedia models. These included incorrect event labels, a bias towards popular YouTube users (where a single user might upload many videos for one event), inaccessible videos, and videos incorrectly categorized into multiple events. These problems could lead to models that learned spurious correlations or failed to generalize well. This paper aims to address these shortcomings by systematically refining the EventNet dataset to create a higher-quality, more reliable resource for research.

Methods

The authors implemented a multi-pronged approach to refine the EventNet dataset:

  • Manual Verification: A team of annotators manually reviewed video labels to correct or remove mislabeled instances.
  • YouTube User Bias Removal: To prevent models from learning user-specific patterns rather than event characteristics, they limited the number of videos from any single YouTube user per event to a maximum of three.
  • Inaccessible Video Removal: Videos that were no longer available on YouTube were removed to ensure data integrity.
  • Duplicate Event Category Removal: Videos assigned to multiple event categories were removed to maintain a clear, single label for each video.
  • Video Corpus Augmentation: For events that fell below a minimum threshold of 50 videos after the cleaning process, new videos were crawled from YouTube to meet this requirement.

Beyond dataset refinement, they also trained two types of classification models:

  • Convolutional Neural Network (CNN) for Event Classification: They fine-tuned an AlexNet model, pre-trained on ImageNet, by replacing the final layer with 500 output nodes corresponding to the events. A background class was added, and data from ActivityNet was used to supplement the training set.
  • Linear SVM for Event-Specific Concepts: For each event-specific concept, they trained a binary linear SVM classifier. This involved extracting features from video frames and using positive samples from the target concept and negative samples from other events.

Datasets

  • EventNet v1.1: This is the primary dataset introduced in the paper. It contains 67,641 videos, 500 events, and 5,028 event-specific concepts. It's an updated and cleaned version of the original EventNet.
  • Original EventNet: Used as a baseline for comparison, it contained 99,382 videos.
  • ImageNet: Used for pre-training the AlexNet CNN model.
  • ActivityNet: Used to supplement the training data for the CNN event classification model, specifically for creating a background class and ensuring semantic diversity.

Key Insights and Ideas

  • Data Quality is Paramount: The paper strongly emphasizes that even with a large dataset, poor data quality (noise, bias) can severely limit model performance. The meticulous cleaning process highlights the importance of dataset curation.
  • YouTube User Bias is a Real Problem: The finding that limiting videos per user significantly impacts the dataset size and diversity underscores a critical challenge in using large-scale web-crawled video data. Models trained on such data might overfit to specific uploaders' styles.
  • Augmentation is Necessary for Balanced Datasets: After aggressive cleaning, some events were left with too few videos. The proactive approach of crawling more videos to ensure a minimum of 50 per event is crucial for balanced training.
  • CNNs Outperform SVMs for Multi-Class Event Classification: The results suggest that CNNs are better suited for the complex multi-class event classification task, especially when dealing with a large number of classes and potentially limited negative samples for SVMs.
  • Binary SVMs are Effective for Concept Detection: Linear SVMs proved to be a viable approach for detecting specific event-related concepts, likely due to the more focused nature of binary classification.

Key Results

  • EventNet v1.1 Size: The refined dataset contains 67,641 videos, 500 events, and 5,028 event-specific concepts.
  • Video Removal:
    • 10,521 videos were removed due to incorrect event labels.
  • 17,257 videos were removed to mitigate YouTube user bias.
  • 6,130 videos were removed because they were inaccessible.
  • Classification Performance (Test Set Accuracies):
    • SVM (5K negative samples): Top-1: 0.1842, Top-5: 0.3809 (binary accuracy: 0.6342)
    • SVM (20K negative samples): Top-1: 0.1933, Top-5: 0.3799 (binary accuracy: 0.7422)
    • CNN: Top-1: 0.3067, Top-5: 0.5327

The CNN model achieved significantly higher Top-1 and Top-5 accuracies for multi-class event classification compared to SVMs. The binary SVMs showed strong performance, with the version using 20K negative samples achieving 0.7422 accuracy.

Limitations and Future Work

  • Ongoing Data Volatility: The authors acknowledge that videos are constantly being uploaded and removed from YouTube, meaning the dataset will continue to require updates to remain current. They have stored the data and metadata to facilitate this.
  • Potential for New Biases: While user bias was addressed, other subtle biases (e.g., related to video quality, production style, or specific regions) might still exist and are not explicitly discussed as being removed.
  • Future Work: The paper states that EventNet v1.1 will be made publicly available, which will facilitate further research. The improved dataset and trained models are expected to significantly advance research in computer vision and multimedia. The authors also mention having a separate version of the corpus that retains repeated videos for potential future experiments.

Summary updated

Abstract

EventNet is a large-scale video corpus and event ontology consisting of 500 events associated with event-specific concepts. In order to improve the quality of the current EventNet, we conduct the following steps and introduce EventNet version 1.1: (1) manually verify the correctness of event labels for all videos; (2) remove the YouTube user bias by limiting the maximum number of videos in each event from the same YouTube user as 3; (3) remove the videos which are currently not accessible online; (4) remove the video belonging to multiple event categories. After the above procedure, some events may contain only a small number of videos, and therefore we crawl more videos for those events to ensure every event will contain more than 50 videos. Finally, EventNet version 1.1 contains 67,641 videos, 500 events, and 5,028 event-specific concepts. In addition, we train a Convolutional Neural Network (CNN) model for event classification via fine-tuning AlexNet using EventNet version 1.1. Then we use the trained CNN model to extract FC7 layer feature and train binary classifiers using linear SVM for each event-specific concept. We believe this new version of EventNet will significantly facilitate research in computer vision and multimedia, and will put it online for public downloading in the future.
BibTeX
@misc{wang2016eventnet,
  author = {Dongang Wang and Zheng Shou and Hongyi Liu and Shih-Fu Chang},
  title = {{EventNet Version 1.1 Technical Report}},
  year = {2016},
  eprint = {1605.07289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1605.07289},
}
Figure