Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning

Baoyuan Wu, Weidong Chen, Yanbo Fan, Yong Zhang, Jinlong Hou, Jie Liu, Tong Zhang

Abstract

Tencent researchers have released Tencent ML-Images, a massive multi-label dataset featuring 18 million images and 11,000 categories. By training a ResNet-101 model on this multi-label data rather than traditional single-label datasets like ImageNet, they achieved superior visual representations that significantly boost performance across downstream tasks like object detection and semantic segmentation.

Introduction

Most deep learning models for computer vision are pre-trained on datasets where each image has exactly one label, such as the standard ImageNet-1k. However, natural images are rarely so simple; an image of a "cow" almost always contains "grass," "sky," or "fences."

Forcing a model to learn only a single tag leads to two major issues:

  1. Information Waste: Useful visual features (like the texture of the grass) are ignored or treated as noise.
  2. Model Confusion: If two visually similar images containing both a cow and grass are labeled differently—one as "cow" and one as "grass"—the model receives contradictory signals during training.

Multi-label pre-training is proposed as the key to learning more robust, generalized visual representations that better reflect the complexity of the real world.

The Tencent ML-Images Dataset

The authors constructed the Tencent ML-Images dataset by merging and refining two existing large-scale datasets: ImageNet-11k and Open Images. The resulting dataset contains 17.6 million training images and 88,000 validation images across 11,166 unique categories.

Distribution of annotated tags per image in the Tencent ML-Images dataset.

As shown in Figure 6, the dataset is densely annotated, with a significant portion of images containing between 5 and 15 tags. This multi-label nature allows models to learn richer feature representations compared to single-label benchmarks.

Methodology

To effectively utilize this massive dataset, several novel techniques were employed:

Tag Augmentation and Hierarchy

The WordNet semantic hierarchy was used to automatically add "ancestor" tags (e.g., a "Husky" label automatically triggers "Dog" and "Animal"). Additionally, a model trained on Open Images predicted extra relevant tags for ImageNet images to fill gaps in the original annotations.

Handling Class Imbalance

With over 11,000 categories, the dataset exhibits extreme class imbalance. Some tags appear millions of times, while others appear only hundreds of times.

Log-scale distribution of images per class index, highlighting the long-tail nature of the dataset.

To address the distribution shown in Figure 5, the authors designed a weighted cross-entropy loss. This included a cost parameter ($\eta=12$) to penalize missing positive labels more heavily and an adaptive weight ($r_t$) that decays the importance of a category if it appears too frequently in sequential mini-batches, effectively "slowing down" learning for common objects.

Distributed Training (TFplus)

Training on 18M images is computationally intensive. The authors utilized TFplus, a custom framework integrating MPI and NVIDIA’s NCCL for optimized communication. This setup allowed a ResNet-101 to be trained for 60 epochs in just 90 hours using 128 GPUs, achieving 86% scaling efficiency.

Experimental Results

The effectiveness of Tencent ML-Images was evaluated through transfer learning on several benchmarks:

  • ImageNet-1k Classification: A ResNet-101 pre-trained on Tencent ML-Images reached 80.73% top-1 accuracy after fine-tuning, outperforming the baseline trained from scratch (77.8%) and surpassing Google’s JFT-300M pre-trained model (79.2%) under similar conditions.
  • Object Detection: On PASCAL VOC 2007, the model achieved 81.5% mAP, a significant improvement over the 76.3% achieved by standard ImageNet pre-training.
  • Semantic Segmentation: On PASCAL VOC 2012, it reached 76.3% mIOU, confirming the high quality of the learned features for pixel-level tasks.

Conclusion and Future Work

Tencent ML-Images demonstrates that teaching a model to recognize multiple entities within an image produces denser, more transferable visual features. While the dataset faces challenges like label noise and a relatively small validation set, its scale and multi-label nature provide a powerful foundation for various vision tasks. Future work may involve combining this dataset with self-supervised learning methods or extending it to tasks like human pose estimation.

The dataset, checkpoints, and training code are available at https://github.com/Tencent/tencent-ml-images.

Summary updated

Abstract

In existing visual representation learning tasks, deep convolutional neural networks (CNNs) are often trained on images annotated with single tag, such as ImageNet. However, single tag annotation cannot describe all important contents of one image, and some useful visual information may be wasted during training. In this work, we propose to train CNNs from images annotated with multiple tags, to enhance the quality of visual representation of the trained CNN model. To this end, we build a large-scale multi-label image database with 18M images and 11K categories, dubbed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Tencent ML-Images</i> . We efficiently train the ResNet-101 model with multi-label outputs on Tencent ML-Images, taking 90 hours for 60 epochs, based on a large-scale distributed deep learning framework, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , TFplus. The good quality of the visual representation of the Tencent ML-Images checkpoint is verified through three transfer learning tasks, including single-label image classification on ImageNet and Caltech-256, object detection on PASCAL VOC 2007, and semantic segmentation on PASCAL VOC 2012. The Tencent ML-Images database, the checkpoints of ResNet-101, and all the training codes have been released at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/Tencent/tencent-ml-images</uri></i> . It is expected to promote other vision tasks in the research and industry community.
BibTeX
@article{wu2019tencent,
  author = {Baoyuan Wu and Weidong Chen and Yanbo Fan and Yong Zhang and Jinlong Hou and Jie Liu and Tong Zhang},
  title = {{Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2956775},
}
Figure