Large-Scale Datasets for Going Deeper in Image Understanding

Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, Yizhou Wang, Yonggang WangICME 2019 · DOI · OpenReview

Introduction

The AI Challenger (AIC) dataset represents a significant open-source initiative designed to transition computer vision from basic object recognition toward comprehensive image understanding. While deep learning has transformed the field, progress is frequently constrained by the scale and nature of existing data. Most datasets prioritize classification or detection, leaving a "semantic gap" between low-level pixel data and high-level human concepts.

The AIC project addresses three critical deficiencies in the current landscape:

  1. Human Pose Estimation: Existing datasets like MSCOCO are often insufficient for the complexities of multi-person keypoint detection.
  2. Zero-Shot Learning: Current attribute datasets often lack "semantic" attributes (e.g., "is a predator"), hindering a model's ability to generalize to unseen objects.
  3. Language Diversity: There is a notable scarcity of high-quality, large-scale Chinese image captioning data compared to English-centric resources.

The AI Challenger Dataset

The AIC dataset is structured into three specialized sub-datasets designed to push the boundaries of pose estimation, attribute recognition, and multi-modal understanding.

Overview of the three AIC sub-datasets: Human Keypoint Detection (HKD), Large-scale Attribute Dataset (LAD), and Image Chinese Captioning (ICC).
  • Human Keypoint Detection (HKD): Comprises 300,000 images featuring 700,000 human instances, each annotated with 14 skeletal keypoints.
  • Large-scale Attribute Dataset (LAD): Includes 81,658 images across 240 classes with 359 attributes. Crucially, it blends visual traits (color, shape) with semantic traits (habit, function).
  • Image Chinese Captioning (ICC): Features 300,000 images accompanied by 1.5 million unique, fluent Chinese captions (five per image).

A key feature of the AIC dataset is the 95% overlap between HKD and ICC images, enabling researchers to explore multi-task learning, such as using pose data to inform more accurate natural language descriptions.

Methodology and Annotation

Creating a dataset of this magnitude required a rigorous multi-stage pipeline involving over 185,000 worker hours.

  • Collection and Cleaning: Images were crawled from search engines and manually filtered to ensure they met the specific requirements for single-object or multi-object scenes.
  • Annotation: For HKD, a hierarchical approach was used, first labeling bounding boxes and then 14 skeletal keypoints with visibility flags. For ICC, native Chinese speakers were employed to ensure linguistic fluency and descriptive accuracy.
  • Baselines: The authors established performance benchmarks using standard architectures, such as SSD/Mask R-CNN for detection and encoder-decoder models for captioning.

Key Results and Insights

The results demonstrate that the sheer scale of the AIC dataset acts as a powerful teacher. Even baseline models showed marked improvement when trained on data of this volume.

A dense visualization of skeletal connections in the HKD dataset, illustrating the immense variety and scale of human poses captured.
  • HKD Performance: The dataset is significantly larger than MSCOCO, providing a much richer training ground. The baseline model achieved a mAP-14 of 0.234, indicating substantial room for future algorithmic innovation.
  • LAD and Semantic Attributes: The inclusion of high-level semantic attributes (e.g., "is dangerous") in LAD mimics human learning patterns. The Matrix Distribution Projection (MDP) method led the zero-shot recognition benchmarks with 48.58% accuracy.
  • ICC and Linguistic Nuance: The ICC baseline achieved a BLEU-1 score of 0.765. Qualitative analysis showed that models trained on ICC produced more context-aware captions than those relying on translated data, highlighting the importance of language-specific resources.

Conclusion and Future Directions

While the AIC dataset provides a massive leap forward, the authors identify several avenues for growth. Future iterations could deepen annotations to include facial expressions or specific action labels. Additionally, expanding the ICC dataset beyond human-centric scenes would improve model robustness. The most promising frontier remains the exploration of joint training across all three tasks, leveraging the dataset's unique overlapping structure to achieve a more holistic form of machine "understanding."

Summary updated

Abstract

Recently, extensive efforts have been devoted to computer vision and machine learning by exploiting big data to explore many practical applications. However, these research fields are still quite limited not only by the sheer volume, but also the versatility and diversity, of the available datasets. In this paper, we target at four challenging and yet important computer vision tasks, namely, human-centered scene classification, attribute based zero-shot learning (recognition), human keypoint detection and image Chinese captioning. Four novel large-scale datasets are collected and annotated to facilitate these tasks of deeper image understanding. Labels, bounding boxes, attributes, keypoints and captions are annotated in corresponding datasets. These rich annotations bridge the semantic gap between low-level images and high-level concepts. Extensive experiments on baseline methods have been implemented and compared, which show that these learning tasks on our datasets are still challenging.
BibTeX
@inproceedings{wu2019large,
  author = {Jiahong Wu and He Zheng and Bo Zhao and Yixin Li and Baoming Yan and Rui Liang and Wenjia Wang and Shipei Zhou and Guosen Lin and Yanwei Fu and Yizhou Wang and Yonggang Wang},
  title = {{Large-Scale Datasets for Going Deeper in Image Understanding}},
  booktitle = {ICME},
  year = {2019},
  doi = {10.1109/icme.2019.00256},
}
Figure