Learning to Detect Concepts from Webly-Labeled Video Data

Junwei Liang, Lu Jiang, Deyu Meng, Alexander G. Hauptmann

Abstract

Training video concept detectors typically requires vast amounts of manually labeled data, which is prohibitively expensive to acquire at scale. This paper introduces WELL (WEbly-Labeled Learning), a framework designed to learn directly from noisy web metadata such as titles and tags. By combining curriculum learning with a novel partial-order approach and sample-level dropout, WELL effectively mitigates the impact of label noise. Experimental results on the FCVID and YFCC100M datasets demonstrate that WELL achieves state-of-the-art performance, proving that large-scale noisy web data can effectively substitute for smaller, manually curated datasets.

Introduction

The "data hunger" of modern computer vision models presents a significant bottleneck for video analysis. While image datasets like ImageNet boast millions of labels, video datasets are often orders of magnitude smaller due to the high cost of manual annotation. For instance, the Fudan-Columbia Video Dataset (FCVID) contains only 0.09 million labels compared to ImageNet’s 14 million.

To address this, we propose leveraging the abundance of "web labels"—user-provided titles, descriptions, and tags. However, web data is notoriously noisy; keywords may be missing (low recall) or irrelevant to the visual content (low precision). The WELL framework provides a theoretically grounded approach to harness this messy data effectively.

Methodology: The WELL Framework

The core of WELL is built upon two pillars: Curriculum Learning (CL) and Self-Paced Learning (SPL). CL mimics human education by presenting "easy" examples before "hard" ones, while SPL allows the model to dynamically adjust its learning pace based on its current state.

Partial-Order Curriculum

Traditional curriculum learning requires a "total order," ranking every video by difficulty—a task that is computationally infeasible at web scale. WELL introduces a partial-order curriculum, where videos are grouped into broad categories based on metadata confidence (e.g., "keyword in title" vs. "keyword in tags"). The model prioritizes confident groups without requiring a strict internal ranking.

Sample-Level Dropout

To prevent the model from overfitting to specific noisy labels early in training, we introduce sample-level dropout. Similar to neural network dropout, this technique randomly ignores subsets of samples during iterations, encouraging the model to learn more robust, generalizable visual patterns rather than memorizing noise.

Experimental Results

We evaluated WELL on two massive benchmarks: FCVID (91,223 videos) and YFCC100M (~800,000 videos). In both cases, WELL significantly outperformed existing baselines.

Table 1: Performance comparison on the FCVID dataset. WELL (binary) achieves the highest mAP, significantly exceeding standard BatchTrain and specialized methods like BabyLearning.

On the FCVID dataset, WELL achieved a Mean Average Precision (mAP) of 0.567, a substantial improvement over BatchTrain (0.469) and BabyLearning (0.496). On the YFCC100M dataset, WELL maintained high precision (P@10 of 0.649), outperforming Google’s hard negative mining approach (0.500).

Discussion: Quantity vs. Quality

A key insight from this research is the trade-off between data volume and label quality. Our experiments show that a model trained on a large volume of noisy data can surpass one trained on a smaller set of clean data.

Table 2: Comparison of WELL (trained on noisy web data) vs. BatchTrain* (trained on clean manual labels). Note that 2,000 hours of noisy data (0.567) outperforms 500 hours of clean data (0.561).

As shown in the table above, WELL trained on 2,000 hours of noisy web data achieved a performance of 0.567, which is superior to the 0.561 achieved by BatchTrain using 500 hours of manually labeled "clean" data. This suggests that for video concept detection, scale can compensate for noise when using a robust learning framework.

Conclusion and Future Work

WELL demonstrates that webly-labeled data is a viable and powerful resource for training video detectors. By formalizing the "easy-to-hard" heuristic as a non-convex optimization problem, we provide a solid theoretical foundation for large-scale web learning. Future work will focus on automating the "early stopping" process to remove the need for any manual validation and integrating WELL into end-to-end deep learning architectures for joint feature and detector optimization.

Summary updated

Abstract

Learning detectors that can recognize concepts, such as people actions, objects, etc., in video content is an interesting but challenging problem. In this paper, we study the problem of automatically learning detectors from the big video data on the web without any additional manual annotations. The contextual information available on the web provides noisy labels to the video content. To leverage the noisy web labels, we propose a novel method called WEbly-Labeled Learning (WELL). It is established on two theories called curriculum learning and self-paced learning and exhibits useful properties that can be theoretically verified. We provide compelling insights on the latent non-convex robust loss that is being minimized on the noisy data. In addition, we propose two novel techniques that not only enable WELL to be applied to big data but also lead to more accurate results. The efficacy and the scalability of WELL have been extensively demonstrated on two public benchmarks, including the largest multimedia dataset and the largest manually-labeled video set. Experimental results show that WELL significantly outperforms the state-of-the-art methods. To the best of our knowledge, WELL achieves by far the best reported performance on these two webly-labeled big video datasets.
BibTeX
@inproceedings{liang2016learning,
  author = {Junwei Liang and Lu Jiang and Deyu Meng and Alexander G. Hauptmann},
  title = {{Learning to Detect Concepts from Webly-Labeled Video Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure