TL;DR
This paper presents EventNet v1.1, a significantly improved version of a large-scale video corpus for event recognition. By meticulously cleaning and augmenting the dataset, they've created a more robust resource that addresses common biases and data quality issues, making it a better foundation for training advanced event classification models.
What Problem Does This Paper Address?
The original EventNet dataset, while large, suffered from several quality issues that hindered its effectiveness for training computer vision and multimedia models. These included incorrect event labels, a bias towards popular YouTube users (where a single user might upload many videos for one event), inaccessible videos, and videos incorrectly categorized into multiple events. These problems could lead to models that learned spurious correlations or failed to generalize well. This paper aims to address these shortcomings by systematically refining the EventNet dataset to create a higher-quality, more reliable resource for research.
Methods
The authors implemented a multi-pronged approach to refine the EventNet dataset:
- Manual Verification: A team of annotators manually reviewed video labels to correct or remove mislabeled instances.
- YouTube User Bias Removal: To prevent models from learning user-specific patterns rather than event characteristics, they limited the number of videos from any single YouTube user per event to a maximum of three.
- Inaccessible Video Removal: Videos that were no longer available on YouTube were removed to ensure data integrity.
- Duplicate Event Category Removal: Videos assigned to multiple event categories were removed to maintain a clear, single label for each video.
- Video Corpus Augmentation: For events that fell below a minimum threshold of 50 videos after the cleaning process, new videos were crawled from YouTube to meet this requirement.
Beyond dataset refinement, they also trained two types of classification models:
- Convolutional Neural Network (CNN) for Event Classification: They fine-tuned an AlexNet model, pre-trained on ImageNet, by replacing the final layer with 500 output nodes corresponding to the events. A background class was added, and data from ActivityNet was used to supplement the training set.
- Linear SVM for Event-Specific Concepts: For each event-specific concept, they trained a binary linear SVM classifier. This involved extracting features from video frames and using positive samples from the target concept and negative samples from other events.
Datasets
- EventNet v1.1: This is the primary dataset introduced in the paper. It contains 67,641 videos, 500 events, and 5,028 event-specific concepts. It's an updated and cleaned version of the original EventNet.
- Original EventNet: Used as a baseline for comparison, it contained 99,382 videos.
- ImageNet: Used for pre-training the AlexNet CNN model.
- ActivityNet: Used to supplement the training data for the CNN event classification model, specifically for creating a background class and ensuring semantic diversity.
Key Insights and Ideas
- Data Quality is Paramount: The paper strongly emphasizes that even with a large dataset, poor data quality (noise, bias) can severely limit model performance. The meticulous cleaning process highlights the importance of dataset curation.
- YouTube User Bias is a Real Problem: The finding that limiting videos per user significantly impacts the dataset size and diversity underscores a critical challenge in using large-scale web-crawled video data. Models trained on such data might overfit to specific uploaders' styles.
- Augmentation is Necessary for Balanced Datasets: After aggressive cleaning, some events were left with too few videos. The proactive approach of crawling more videos to ensure a minimum of 50 per event is crucial for balanced training.
- CNNs Outperform SVMs for Multi-Class Event Classification: The results suggest that CNNs are better suited for the complex multi-class event classification task, especially when dealing with a large number of classes and potentially limited negative samples for SVMs.
- Binary SVMs are Effective for Concept Detection: Linear SVMs proved to be a viable approach for detecting specific event-related concepts, likely due to the more focused nature of binary classification.
Key Results
- EventNet v1.1 Size: The refined dataset contains 67,641 videos, 500 events, and 5,028 event-specific concepts.
- Video Removal:
- 10,521 videos were removed due to incorrect event labels.
- 17,257 videos were removed to mitigate YouTube user bias.
- 6,130 videos were removed because they were inaccessible.
- Classification Performance (Test Set Accuracies):
- SVM (5K negative samples): Top-1: 0.1842, Top-5: 0.3809 (binary accuracy: 0.6342)
- SVM (20K negative samples): Top-1: 0.1933, Top-5: 0.3799 (binary accuracy: 0.7422)
- CNN: Top-1: 0.3067, Top-5: 0.5327
The CNN model achieved significantly higher Top-1 and Top-5 accuracies for multi-class event classification compared to SVMs. The binary SVMs showed strong performance, with the version using 20K negative samples achieving 0.7422 accuracy.
Limitations and Future Work
- Ongoing Data Volatility: The authors acknowledge that videos are constantly being uploaded and removed from YouTube, meaning the dataset will continue to require updates to remain current. They have stored the data and metadata to facilitate this.
- Potential for New Biases: While user bias was addressed, other subtle biases (e.g., related to video quality, production style, or specific regions) might still exist and are not explicitly discussed as being removed.
- Future Work: The paper states that EventNet v1.1 will be made publicly available, which will facilitate further research. The improved dataset and trained models are expected to significantly advance research in computer vision and multimedia. The authors also mention having a separate version of the corpus that retains repeated videos for potential future experiments.
Summary updated