Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning

Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, Junzhou Huang, Wei Liu, Tong Zhang

TL;DR

This paper introduces Tencent ML-Images, a massive multi-label image dataset with 18 million images and 11,000 categories, designed to improve visual representation learning by capturing richer image content than single-label datasets. They demonstrate its effectiveness by training a ResNet-101 model and showing superior performance on downstream tasks like image classification, object detection, and semantic segmentation compared to models trained on single-label datasets.

What Problem Does This Paper Address?

Current state-of-the-art deep learning models for visual representation learning, like those trained on ImageNet, primarily rely on single-label annotations. This approach is a significant simplification, as most natural images contain multiple objects and concepts. A single label can only capture a fraction of the image's content, leading to wasted visual information during training and potentially suboptimal representations. This limitation is particularly problematic for tasks that require a nuanced understanding of an image's composition. The authors aim to address this by creating a large-scale dataset with multi-label annotations to enable more comprehensive visual representation learning.

Methods

The core of this work is the creation of the Tencent ML-Images dataset. This dataset was built by merging and processing images from existing large-scale datasets like Open Images and ImageNet. Key steps included:

  • Unified Vocabulary: Merging class vocabularies from source datasets and removing abstract or redundant categories to create a unified set of 11,166 categories.
  • Tag Augmentation: Since many source images were single-labeled, the authors developed a method to augment tags. This involved leveraging a semantic hierarchy derived from WordNet and analyzing category co-occurrence patterns to infer additional relevant labels for images.
  • Multi-Label Training: They trained a ResNet-101 model with multi-label outputs on this dataset. To handle the inherent class imbalance and long training times, they designed a novel weighted cross-entropy loss function and utilized a large-scale distributed deep learning framework (TFplus) with MPI and NCCL for efficient training.

Datasets

  • Tencent ML-Images (New Dataset): 18 million images and 11,166 categories. This dataset is a significant contribution, being the largest publicly available multi-label image dataset at the time of publication. It was constructed by combining data from:
  • ImageNet: Specifically, a subset called ImageNet-11k, from which images and categories were filtered and augmented.
  • Open Images: Used to supplement ImageNet data and categories, with filtering and merging to create a unified vocabulary.
  • ImageNet: Used for evaluating transfer learning performance in single-label image classification.
  • Caltech-256: Another benchmark for evaluating transfer learning in single-label image classification.
  • PASCAL VOC 2007 & 2012: Used for evaluating transfer learning in object detection.
  • PASCAL VOC 2012: Used for evaluating transfer learning in semantic segmentation.

Key Insights and Ideas

  • Multi-label is Crucial for Richer Representations: The fundamental idea is that single-label annotations are insufficient for capturing the full semantic content of images, leading to wasted training signals. Multi-label annotations provide a more accurate and comprehensive description, enabling models to learn more robust and generalizable visual features.
  • Tag Augmentation is Key for Large-Scale Multi-Label Datasets: Manually annotating millions of images with multiple labels is prohibitively expensive. The paper demonstrates that leveraging semantic hierarchies and co-occurrence statistics can effectively augment single labels to create a richer multi-label dataset.
  • Specialized Loss Functions are Needed for Imbalance: Large-scale multi-label datasets inherently suffer from severe class imbalance (both between categories and between positive/negative labels within an image). The proposed weighted cross-entropy loss function is critical for effectively training models on such imbalanced data.
  • Distributed Training is Essential for Scalability: Training on 18 million images requires significant computational resources. The paper highlights the importance of efficient distributed training frameworks (like TFplus with MPI/NCCL) to make such large-scale training feasible within a reasonable timeframe.
  • Adaptive Fine-Tuning is Key for Transfer Learning: When transferring models trained on multi-label datasets to single-label tasks, a simple fine-tuning approach can be detrimental. The paper shows that layer-wise adaptive learning rates are crucial for effectively leveraging the learned representations and mitigating the differences between the source and target domains.

Key Results

  • Tencent ML-Images Dataset: A new, large-scale multi-label dataset with 18M images and 11K categories, claimed to be the largest publicly available at the time.
  • Efficient Training: A ResNet-101 model was trained on Tencent ML-Images in 90 hours using 128 GPUs, demonstrating the efficiency of their distributed training setup.
  • Improved Transfer Learning:
    • ImageNet Classification: Fine-tuned models achieved higher top-1 accuracy (e.g., 80.73% with adaptive fine-tuning and 299x299 validation size) compared to baseline ImageNet-trained models and even competitive with models trained on the massive JFT-300M dataset.
  • Caltech-256 Classification: Adaptive fine-tuning from the Tencent ML-Images checkpoint achieved 86.5% accuracy, outperforming a standard ImageNet-trained baseline.
  • Object Detection (PASCAL VOC 2007): Achieved 81.5% [email protected] with their best checkpoint, competitive with models trained on JFT-300M.
  • Semantic Segmentation (PASCAL VOC 2012): Achieved 76.3% mIOU with their best checkpoint, also competitive with JFT-300M-based models.

Limitations and Future Work

  • Validation Set Reliability: The authors note that the validation set for Tencent ML-Images is relatively small (about 1/200th of the training set) and may contain missing labels, making its evaluation metrics less reliable for measuring intrinsic performance. They emphasize that transfer learning is a better indicator of the dataset's quality.
  • Noisy vs. Missing Labels: While they augment tags to reduce missing labels, they acknowledge that noisy tags are unavoidable. They chose a strategy of more missing but less noisy annotations, suggesting that the trade-off between these two is an area for further research.
  • Domain Gap: The paper highlights that the significant differences in data distribution and tasks between Tencent ML-Images and downstream tasks (like VOC) require careful fine-tuning strategies.
  • Future Work: The authors hope their dataset and released code will promote further research in visual representation learning and other vision tasks. The release of the dataset, checkpoints, and code is a significant step towards enabling this.

Summary updated

Abstract

In existing visual representation learning tasks, deep convolutional neural networks (CNNs) are often trained on images annotated with single tags, such as ImageNet. However, a single tag cannot describe all important contents of one image, and some useful visual information may be wasted during training. In this work, we propose to train CNNs from images annotated with multiple tags, to enhance the quality of visual representation of the trained CNN model. To this end, we build a large-scale multi-label image database with 18M images and 11K categories, dubbed Tencent ML-Images. We efficiently train the ResNet-101 model with multi-label outputs on Tencent ML-Images, taking 90 hours for 60 epochs, based on a large-scale distributed deep learning framework,i.e.,TFplus. The good quality of the visual representation of the Tencent ML-Images checkpoint is verified through three transfer learning tasks, including single-label image classification on ImageNet and Caltech-256, object detection on PASCAL VOC 2007, and semantic segmentation on PASCAL VOC 2012. The Tencent ML-Images database, the checkpoints of ResNet-101, and all the training codehave been released at https://github.com/Tencent/tencent-ml-images. It is expected to promote other vision tasks in the research and industry community.
BibTeX
@article{wu2019tencent,
  author = {Baoyuan Wu and Weidong Chen and Yanbo Fan and Yong Zhang and Jinlong Hou and Jie Liu and Junzhou Huang and Wei Liu and Tong Zhang},
  title = {{Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2956775},
}
Figure