Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning

Zhihua Xu, Zhijing Yang, Yufeng Yang, Tianshui Chen

TL;DR

This paper introduces a new benchmark and a novel framework for "webly supervised multi-label recognition" (WS-MLR), a challenging task where images are labeled using noisy web search keywords. Their proposed Dual-Branch Multi-Label Contrastive Learning (DBMLCL) framework effectively tackles label noise by learning category-specific representations and using contrastive learning to correct these noisy labels, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Traditional multi-label image recognition (MLR) relies on meticulously annotated datasets, which are expensive and time-consuming to create, especially for tasks involving many categories. To reduce this burden, researchers have explored "webly supervised learning," using readily available web images and their associated tags or keywords as weak supervision. However, this approach is particularly challenging for multi-label recognition because web keywords are often noisy (false positives and false negatives) and may not cover all objects present in an image. Existing methods for single-label webly supervised learning or for multi-label recognition with clean/partial labels don't directly address the unique noise patterns and complexities of WS-MLR. The lack of a standardized benchmark and fair comparison protocols further hinders progress in this area.

Methods

The core of this paper is the Dual-Branch Multi-Label Contrastive Learning (DBMLCL) framework. It addresses WS-MLR by:

  • Dual-Branch Architecture: Two parallel backbone networks process the input image. This allows for learning diverse representations and encourages robustness.
  • Semantic-Aware Representation Learning: Each branch incorporates a module to extract category-specific instance-level features and maintain category-level prototypes.
  • Instance-Level Contrastive Loss: This loss encourages feature vectors from the two branches for the same image to be similar, while pushing them away from features of negative samples. This helps in learning consistent representations across branches.
  • Prototype Contrastive Loss: This loss aims to make instance-level features closer to their corresponding category prototypes and farther from prototypes of other categories, enhancing category discrimination.
  • Label Correction Module: After an initial warm-up, the model uses prediction scores and feature-prototype similarities to identify and correct noisy labels. This module adaptively adjusts thresholds to refine pseudo-labels.

Datasets

The authors introduce two new datasets for WS-MLR:

  • Web-COCO: Derived from the 80 categories of Microsoft COCO, containing approximately 290,000 images retrieved from the web using keyword combinations. A subset of 20,000 images was manually annotated for evaluation.
  • Web-Pascal: Derived from the 20 categories of Pascal VOC, containing approximately 236,000 images retrieved from the web. A subset of 4,952 images was manually annotated for evaluation.

These datasets are characterized by significant label noise, semantic scattering (objects spread across the image), and class imbalance, making them challenging benchmarks for WS-MLR.

Key Insights and Ideas

  • Webly supervised multi-label recognition is a distinct and challenging problem: The noise inherent in web keywords, combined with the complexity of multiple objects in an image, requires specialized approaches beyond existing MLR or single-label WS methods.
  • Dual-branch learning can enhance robustness: By using two parallel networks, the framework can learn more diverse representations and leverage complementary information, leading to better performance.
  • Contrastive learning is crucial for noise correction: The proposed contrastive losses effectively learn discriminative representations that are essential for identifying and rectifying noisy labels.
  • Category-specific noise correction is more effective than holistic approaches: DBMLCL's focus on learning category-level prototypes and similarities allows for more targeted and accurate label correction.
  • Vision-language models (like CLIP) are helpful but not a complete solution for WS-MLR: While they offer strong priors, they may struggle with the specific noise patterns and lack of explicit supervision found in webly collected multi-label data, highlighting the need for explicit noise modeling.

Key Results

  • State-of-the-art performance: DBMLCL significantly outperforms representative baselines on both Web-COCO and Web-Pascal datasets across various metrics (mAP, OF1, CF1) and data proportions.
  • Web-COCO improvements: DBMLCL achieved the best mAP (71.4%), OF1 (66.6%), and CF1 (66.6%) at 100% data proportion, outperforming the second-best method by a notable margin.
  • Web-Pascal performance: DBMLCL achieved robust performance, obtaining the best CF1 and competitive OF1 scores, demonstrating the importance of explicit noise correction even with strong vision-language priors.
  • Ablation studies confirm component effectiveness: The dual-branch design, instance contrastive loss, and prototype contrastive loss all contribute to the overall performance gains.

Limitations and Future Work

  • Increased computational cost: The dual-branch architecture and contrastive learning introduce higher training computation compared to single-branch methods.
  • Dataset biases: Web-COCO and Web-Pascal may still exhibit search-engine biases, long-tailed category coverage, off-topic images, and language-dependent retrieval noise.
  • Limited integration of vision-language models: The current framework uses vision-language models primarily for comparison, rather than deeply integrating their semantic priors into the noise correction process.

Future work suggested includes investigating computational efficiency, exploring more efficient training strategies, and incorporating stronger vision-language priors for improved WS-MLR.

Summary updated

Abstract

Training deep learning models with freely available web images can reduce their dependence on costly manual annotations. Although webly supervised learning has been widely studied for single-label recognition, its multi-label counterpart remains underexplored, partly due to the lack of unified benchmarks and fair comparison protocols. To address this gap, we construct a benchmark for webly supervised multi-label recognition (WS-MLR), including Web-COCO and Web-Pascal, and re-implement representative baselines under a unified setting. The two datasets cover the same 80 and 20 categories as MS-COCO and Pascal VOC, respectively, and contain about 300 thousand images retrieved from the Internet using category-word combinations as search keywords. We further propose a Dual-Branch Multi-Label Contrastive Learning (DBMLCL) framework, which learns category-specific instance-level and category-level representations together with their similarities to identify and correct noisy labels. Extensive experiments on the benchmark demonstrate that DBMLCL achieves superior performance compared to representative baselines.
BibTeX
@article{xu2026webly,
  author = {Zhihua Xu and Zhijing Yang and Yufeng Yang and Tianshui Chen},
  title = {{Webly Supervised Multi-Label Recognition: Evaluation Benchmark and Dual-Branch Multi-Label Contrastive Learning}},
  journal = {Image and Vision Computing},
  year = {2026},
  doi = {10.1016/j.imavis.2026.106103},
}

Loading related content...

Figure