Web video metadata is frequently imprecise and lacks temporal localization. This paper introduces a non-parametric "lazy learning" framework that utilizes millions of images from web sources like Flickr, Google, and Bing as a reference library. By leveraging this "collective knowledge," the system automatically cleans noisy video-level tags and identifies the specific frames to which they belong, without requiring pre-trained classifiers for every tag.
Introduction: The Metadata Challenge
Searching for specific moments in web videos is often hindered by two major metadata issues: imprecision (noisy or overly general tags) and a lack of localization (tags not linked to specific timestamps). Traditional computer vision relies on training fixed "concept detectors," which cannot scale to the internet's open and rapidly evolving vocabulary. This research proposes a data-driven method to bridge the "semantic gap" using real-time web retrieval.
The tag refinement and localization process: the system filters irrelevant tags (crossed out) for specific frames while retaining relevant ones for others.
The proposed framework operates on a "lazy learning" principle, performing the primary computation at test time rather than during a pre-training phase. The process involves four key steps:
Retrieval Set Construction: The system crawls images from Google, Bing, and Flickr based on the video's initial tags to create a tailored knowledge pool.
Visual Neighbor Voting: Keyframes are compared against this pool to find the $K$ most visually similar neighbors using bag-of-visual-words descriptors.
Tag Relevance Scoring: A modified voting algorithm identifies relevant tags by their frequency among neighbors relative to the overall collection.
Temporal Smoothing: A Gaussian weighting function ensures that tag assignments are consistent across continuous video segments.
Experimental Evaluation and Insights
The system was tested on the DUT-WEBV dataset and an "in the wild" set of YouTube videos.
Iconic images retrieved from web search engines provide a clear visual reference for refining video tags.
Superior Precision: The method achieved an average precision of 65.3%, outperforming the MIL-BPNET baseline of 55.3%.
Source Synergy: Combining Video, Bing, Google, and Flickr (V+B+G+F) yielded the highest accuracy. The study found that different sources provide complementary perspectives—Flickr offers artistic views, while Google and Bing provide "iconic" or objective representations.
Efficiency: The implementation processes video frames in approximately 0.17 seconds, demonstrating viability for large-scale use.
Conclusion and Future Work
This data-driven approach demonstrates that "lazy learning" from web-scale image collections is an effective strategy for refining open-vocabulary video tags. While the system is highly precise, future research will focus on improving recall and mitigating "semantic drift" caused by synonym-based retrieval.
Summary updated
Abstract
Highlights • Our approach locates the temporal positions of tags in videos at the keyframe level. • We deal with a scenario in which there is no pre-defined set of tags. • We report experiments about the use of different web sources (Flickr, Google, Bing). • We show state-of-the-art results on DUT-WEBV, a large dataset of YouTube videos. • We show results in a real-world scenario to perform open vocabulary tag annotation. Abstract Tagging of visual content is becoming more and more widespread as web-based services and social networks have popularized tagging functionalities among their users. These user-generated tags are used to ease browsing and exploration of media collections, e.g. using tag clouds, or to retrieve multimedia content. However, not all media are equally tagged by users. Using the current systems is easy to tag a single photo, and even tagging a part of a photo, like a face, has become common in sites like Flickr and Facebook. On the other hand, tagging a video sequence is more complicated and time consuming, so that users just tag the overall content of a video. In this paper we present a method for automatic video annotation that increases the number of tags originally provided by users, and localizes them temporally, associating tags to keyframes. Our approach exploits collective knowledge embedded in user-generated tags and web sources, and visual similarity of keyframes and images uploaded to social sites like YouTube and Flickr, as well as web sources like Google and Bing. Given a keyframe, our method is able to select “on the fly” from these visual sources the training exemplars that should be the most relevant for this test sample, and proceeds to transfer labels across similar images. Compared to existing video tagging approaches that require training classifiers for each tag, our system has few parameters, is easy to implement and can deal with an open vocabulary scenario. We demonstrate the approach on tag refinement and localization on DUT-WEBV, a large dataset of web videos, and show state-of-the-art results.
BibTeX
@article{ballan2015data,
author = {Lamberto Ballan and Marco Bertini and Giuseppe Serra and Alberto Del Bimbo},
title = {{A data-driven approach for tag refinement and localization in web videos}},
journal = {Comput. Vis. Image Underst.},
year = {2015},
doi = {10.1016/j.cviu.2015.05.009},
}