TagBook: A Semantic Video Representation Without Supervision for Event Detection

Masoud Mazloom, Xirong Li, Cees G. M. Snoek

TL;DR

This paper introduces TagBook, a novel semantic video representation that leverages freely available social media tags to detect events, even when training examples are scarce or non-existent. By propagating tags from visually similar videos and refining the tag assignment process, TagBook achieves state-of-the-art performance in few-example and zero-example event detection scenarios.

What Problem Does This Paper Address?

Event detection in videos is a challenging task, especially when you have very few or no labeled examples of the event you want to detect. Traditional methods often rely on pre-trained concept detectors, which require extensive labeled data for training those detectors themselves. This creates a "chicken and the egg" problem: you need lots of data to train detectors, but you need those detectors to build a representation for event detection when data is scarce. The authors aim to overcome this by creating a semantic video representation that doesn't require training intermediate concept detectors, instead relying on readily available social media tags.

Methods

The core of the TagBook approach is a tag propagation algorithm inspired by image retrieval techniques. Here's a breakdown of the key components:

  • Video-Level Tag Propagation: Instead of propagating tags frame-by-frame (which can be computationally intensive), TagBook operates at the video level. This makes it more scalable for large video datasets.
  • Socially Tagged Videos as a Source: The system utilizes a large collection of publicly available, user-tagged videos (like those from YouTube) as its knowledge base.
  • Nearest Neighbor Tagging: For a given video, the algorithm finds visually similar videos within the source set. Tags from these "neighbors" are then propagated to the target video.
  • Source Set Refinement: To combat the inherent noise and inaccuracy of social tags, the authors introduce a refinement step. This involves improving the quality of tags associated with the source videos before they are used for propagation.
  • Tag Assignment Variants: Three variants of TagBook are proposed:
  • TagBook-hard: Uses binary weights for neighbors and raw tags.
  • TagBook-soft: Weights neighbors based on visual similarity and uses raw tags.
  • TagBook-refine: Weights neighbors by similarity and uses refined tags from the source set. TagBook-refine is shown to be the most effective.
  • Event Representation: Events themselves are represented as tag vectors, allowing for similarity computation with video tag vectors using cosine similarity.

Datasets

The proposed TagBook approach was evaluated on several datasets:

  • TRECVID MED 2013 and 2014: These are standard benchmarks for multimedia event detection, containing user-generated web videos with video-level annotations for event categories. The authors used these for both training (source set) and testing.
  • Columbia Consumer Video (CCV) dataset: This dataset also contains user-generated videos and was used for evaluation.
  • VideoStory46K dataset: This dataset was used as the source set for TagBook construction, providing 46,000 YouTube videos with captions.

Key Insights and Ideas

  • Leveraging Social Data is Powerful: The paper demonstrates that freely available social media tags, despite their potential for noise and ambiguity, can form a surprisingly effective basis for semantic video representation.
  • Refinement is Crucial: Simply propagating raw social tags is less effective than first refining the source set's tags. This highlights the importance of cleaning and improving the quality of the training data.
  • Video-Level Representation is Scalable: Operating at the video level rather than frame level significantly improves the efficiency and scalability of the tag propagation process.
  • Model-Free Approaches Shine: The authors suggest that model-free tag propagation methods (like neighbor voting) are more robust to the subjective and noisy nature of social tags compared to model-based approaches.

Key Results

  • Superior Performance: TagBook, particularly the TagBook-refine variant, consistently outperformed other state-of-the-art semantic video representations on the TRECVID MED 2013 and 2014 datasets for both few-example and zero-example event detection.
  • Zero-Example Success: TagBook achieved a Mean Average Precision (MAP) of 0.198 on TRECVID MED 2013 for few-example detection and 0.091 for zero-example detection, outperforming other methods.
  • Compact Representation: The authors found that using a reduced TagBook size (around 2,000-2,500 tags) yielded optimal performance, suggesting a compact yet descriptive representation.
  • Accurate Video Descriptions: TagBook-refine was also shown to generate more accurate video descriptions compared to other variants, demonstrating its descriptive power.

Limitations and Future Work

  • Tag Ambiguity: While TagBook mitigates issues with social tags, the inherent ambiguity and incompleteness of these tags remain a challenge.
  • Computational Cost: While more scalable than frame-level methods, tag propagation and source set refinement still involve significant computation. The paper notes that non-linear kernels improve accuracy but drastically increase computation time.
  • Future Work: The authors suggest that TagBook could benefit from more discriminative and multimodal representations, and that it is orthogonal to other promising approaches like pseudo-relevance feedback. They also hint at exploring TagBook's potential for other tasks beyond event detection.

Summary updated

Abstract

We consider the problem of event detection in video for scenarios where only a few, or even zero, examples are available for training. For this challenging setting, the prevailing solutions in the literature rely on a semantic video representation obtained from thousands of pretrained concept detectors. Different from existing work, we propose a new semantic video representation that is based on freely available social tagged videos only, without the need for training any intermediate concept detectors. We introduce a simple algorithm that propagates tags from a video's nearest neighbors, similar in spirit to the ones used for image retrieval, but redesign it for video event detection by including video source set refinement and varying the video tag assignment. We call our approach TagBook and study its construction, descriptiveness, and detection performance on the TRECVID 2013 and 2014 multimedia event detection datasets and the Columbia Consumer Video dataset. Despite its simple nature, the proposed TagBook video representation is remarkably effective for few-example and zero-example event detection, even outperforming very recent state-of-the-art alternatives building on supervised representations.
BibTeX
@article{mazloom2016tagbook,
  author = {Masoud Mazloom and Xirong Li and Cees G. M. Snoek},
  title = {{TagBook: A Semantic Video Representation Without Supervision for Event Detection}},
  journal = {IEEE Trans. Multimedia},
  year = {2016},
  url = {https://openreview.net/forum?id=hp-UuLAwoxr},
}
Figure