Abstract
Tencent researchers have released Tencent ML-Images, a massive multi-label dataset featuring 18 million images and 11,000 categories. By training a ResNet-101 model on this multi-label data rather than traditional single-label datasets like ImageNet, they achieved superior visual representations that significantly boost performance across downstream tasks like object detection and semantic segmentation.
Introduction
Most deep learning models for computer vision are pre-trained on datasets where each image has exactly one label, such as the standard ImageNet-1k. However, natural images are rarely so simple; an image of a "cow" almost always contains "grass," "sky," or "fences."
Forcing a model to learn only a single tag leads to two major issues:
- Information Waste: Useful visual features (like the texture of the grass) are ignored or treated as noise.
- Model Confusion: If two visually similar images containing both a cow and grass are labeled differently—one as "cow" and one as "grass"—the model receives contradictory signals during training.
Multi-label pre-training is proposed as the key to learning more robust, generalized visual representations that better reflect the complexity of the real world.
The Tencent ML-Images Dataset
The authors constructed the Tencent ML-Images dataset by merging and refining two existing large-scale datasets: ImageNet-11k and Open Images. The resulting dataset contains 17.6 million training images and 88,000 validation images across 11,166 unique categories.
As shown in Figure 6, the dataset is densely annotated, with a significant portion of images containing between 5 and 15 tags. This multi-label nature allows models to learn richer feature representations compared to single-label benchmarks.
Methodology
To effectively utilize this massive dataset, several novel techniques were employed:
Tag Augmentation and Hierarchy
The WordNet semantic hierarchy was used to automatically add "ancestor" tags (e.g., a "Husky" label automatically triggers "Dog" and "Animal"). Additionally, a model trained on Open Images predicted extra relevant tags for ImageNet images to fill gaps in the original annotations.
Handling Class Imbalance
With over 11,000 categories, the dataset exhibits extreme class imbalance. Some tags appear millions of times, while others appear only hundreds of times.
To address the distribution shown in Figure 5, the authors designed a weighted cross-entropy loss. This included a cost parameter ($\eta=12$) to penalize missing positive labels more heavily and an adaptive weight ($r_t$) that decays the importance of a category if it appears too frequently in sequential mini-batches, effectively "slowing down" learning for common objects.
Distributed Training (TFplus)
Training on 18M images is computationally intensive. The authors utilized TFplus, a custom framework integrating MPI and NVIDIA’s NCCL for optimized communication. This setup allowed a ResNet-101 to be trained for 60 epochs in just 90 hours using 128 GPUs, achieving 86% scaling efficiency.
Experimental Results
The effectiveness of Tencent ML-Images was evaluated through transfer learning on several benchmarks:
- ImageNet-1k Classification: A ResNet-101 pre-trained on Tencent ML-Images reached 80.73% top-1 accuracy after fine-tuning, outperforming the baseline trained from scratch (77.8%) and surpassing Google’s JFT-300M pre-trained model (79.2%) under similar conditions.
- Object Detection: On PASCAL VOC 2007, the model achieved 81.5% mAP, a significant improvement over the 76.3% achieved by standard ImageNet pre-training.
- Semantic Segmentation: On PASCAL VOC 2012, it reached 76.3% mIOU, confirming the high quality of the learned features for pixel-level tasks.
Conclusion and Future Work
Tencent ML-Images demonstrates that teaching a model to recognize multiple entities within an image produces denser, more transferable visual features. While the dataset faces challenges like label noise and a relatively small validation set, its scale and multi-label nature provide a powerful foundation for various vision tasks. Future work may involve combining this dataset with self-supervised learning methods or extending it to tasks like human pose estimation.
The dataset, checkpoints, and training code are available at https://github.com/Tencent/tencent-ml-images.
Summary updated