Webly Supervised Learning with Category-level Semantic Information

Li Niu, Qingtao TangarXiv 2018 · arXiv

Abstract

As tons of photos are being uploaded to public websites (e.g., Flickr, Bing, and Google) every day, learning from web data has become an increasingly popular research direction because of freely available web resources, which is also referred to as webly supervised learning. Nevertheless, the performance gap between webly supervised learning and traditional supervised learning is still very large, owning to the label noise of web data as well as the domain shift between web data and test data. To be exact, on one hand, the labels of images crawled from public websites are very noisy and often inaccurate. On the other hand, the data distributions between web data and test data are considerably different, which is known as domain shift. Some existing works tend to facilitate learning from web data with the aid of extra information, such as augmenting or purifying web data by virtue of instance-level supervision, which is usually in demand of heavy manual annotation. Instead, we propose to tackle the label noise and domain shift by leveraging more accessible category-level supervision. In particular, we build our method upon variational autoencoder (VAE), in which the classification network is attached on the hidden layer of VAE in a way that the classification network and VAE can jointly leverage the category-level hybrid semantic information. Moreover, we further extend our method to cope with the domain shift by utilizing unlabeled test instances in the training stage followed by low-rank refinement. The effectiveness of our proposed methods is clearly demonstrated by extensive experiments on three benchmark datasets.

Introduction

Deep learning's success is heavily dependent on large-scale, high-quality labeled datasets. However, manual annotation is costly and slow. Webly Supervised Learning (WSL) offers an alternative by training models on images crawled from the web. While abundant, web data is notoriously noisy—search queries often return irrelevant images (e.g., a search for "ox" returning restaurant signs or unrelated objects). Furthermore, a "domain shift" exists between web-crawled images and the curated test sets typically used for evaluation.

Existing WSL approaches often rely on "instance-level" supervision, such as clean subsets or bounding boxes, to mitigate noise. This paper proposes WSCI (Webly Supervised learning with Category-level Information), a framework that leverages more accessible category-level supervision to tackle label noise and domain shift simultaneously.

Methodology: The WSCI Framework

The core of WSCI is the integration of a Variational Autoencoder (VAE) with a standard classification network (CNN).

The WSCI framework architecture: A CNN feature extractor is shared with a Semantic VAE that uses category-level information (A) to regulate the latent space and filter noise.

Semantic VAE as an Outlier Detector

The VAE is designed to learn the underlying distribution of a category. By regulating the latent space with category-level semantic information (matrix A), the VAE acts as a quality filter. If an image is a representative member of a class, the VAE can reconstruct it with low error. Conversely, noisy outliers result in high reconstruction error. During joint training, the VAE assigns a weight to each image based on its reconstruction probability, effectively suppressing the influence of noisy data on the classifier.

Hybrid Semantic Representation

To guide the VAE, the authors utilize a hybrid semantic representation consisting of:

  1. Attributes: Manually defined traits (e.g., "has wings").
  2. Word Vectors: Linguistic embeddings (e.g., GloVe) capturing category relationships.
  3. Visual Encoding: A novel representation generated by summarizing visual features from region proposals across a category using a Gaussian Mixture Model (GMM). This provides a visual "prototype" without requiring manual labeling.

Additionally, WSCI addresses domain shift by incorporating unlabeled test instances during the training stage, followed by a low-rank refinement step to align the web-trained model with the target data distribution.

Experimental Results

The WSCI method was evaluated on three benchmark datasets: AwA2 (animals), CUB (birds), and SUN (scenes). Training data consisted of the top 500 images from Google for each category, with no overlap with test sets.

Quantitative comparison of WSCI against standard CNN and state-of-the-art WSL baselines across three benchmark datasets.

As shown in the table above, WSCI consistently outperforms both basic CNNs and existing state-of-the-art WSL methods. On average, WSCI achieved an accuracy of 70.25%, a significant improvement over the standard CNN baseline (64.06%) and previous methods like Xiao et al. (67.33%). Ablation studies (e.g., WSCI w/o attr) further confirm that both manual attributes and the proposed visual encoding contribute to the model's robustness.

Conclusion

WSCI demonstrates that category-level semantic information is a powerful tool for cleaning noisy web data. By using a VAE as a dynamic quality filter, the framework bridges the gap between webly supervised and fully supervised learning without the need for expensive instance-level annotations. Future work may address specific types of noise, such as "label flip" (misclassification between similar categories), and further optimize the computational efficiency of the visual encoding process.

Summary updated

Abstract

As tons of photos are being uploaded to public websites (e.g., Flickr, Bing, and Google) every day, learning from web data has become an increasingly popular research direction because of freely available web resources, which is also referred to as webly supervised learning. Nevertheless, the performance gap between webly supervised learning and traditional supervised learning is still very large, owning to the label noise of web data as well as the domain shift between web data and test data. To be exact, on one hand, the labels of images crawled from public websites are very noisy and often inaccurate. On the other hand, the data distributions between web data and test data are considerably different, which is known as domain shift. Some existing works tend to facilitate learning from web data with the aid of extra information, such as augmenting or purifying web data by virtue of instance-level supervision, which is usually in demand of heavy manual annotation. Instead, we propose to tackle the label noise and domain shift by leveraging more accessible category-level supervision. In particular, we build our method upon variational autoencoder (VAE), in which the classification network is attached on the hidden layer of VAE in a way that the classification network and VAE can jointly leverage the category-level hybrid semantic information. Moreover, we further extend our method to cope with the domain shift by utilizing unlabeled test instances in the training stage followed by low-rank refinement. The effectiveness of our proposed methods is clearly demonstrated by extensive experiments on three benchmark datasets.
BibTeX
@misc{niu2018webly,
  author = {Li Niu and Qingtao Tang},
  title = {{Webly Supervised Learning with Category-level Semantic Information}},
  year = {2018},
  eprint = {1803.03857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1803.03857},
}
Figure