Abstract: Significant progress has been achieved in Computer Vision by leveraging large-scale image datasets. However, large-scale datasets for complex Computer Vision tasks beyond classification are still limited. This paper proposed a large-scale dataset named AIC (AI Challenger) with three sub-datasets: human keypoint detection (HKD), large-scale attribute dataset (LAD), and image Chinese captioning (ICC). In this dataset, we annotate class labels (LAD), keypoint coordinates (HKD), bounding boxes (HKD and LAD), attributes (LAD), and captions (ICC). These rich annotations bridge the semantic gap between low-level images and high-level concepts. The proposed dataset is an effective benchmark to evaluate and improve different computational methods. In addition, for related tasks, others can also use our dataset as a new resource to pre-train their models.
Introduction
While ImageNet revolutionized computer vision by providing millions of labeled images for classification, understanding a scene goes far beyond just naming the objects in it. To truly "understand" an image, a model needs to grasp human poses, identify fine-grained attributes, and describe complex relationships in natural language. Researchers from Sinovation Ventures and several top Chinese universities introduced AI Challenger (AIC), a massive dataset designed to push the boundaries of image understanding across three distinct but related tasks: human keypoint detection, attribute-based zero-shot recognition, and Chinese image captioning.
Addressing the Semantic Gap
Most existing computer vision datasets focus on simple classification or bounding box detection. While useful, these labels don't capture the depth of human-like image understanding. Specialized datasets for more complex tasks—like human pose estimation or attribute recognition—have historically been small (often under 100k images) or biased. Furthermore, the field of image captioning has been dominated by English-language datasets like MSCOCO, leaving a significant gap for other major languages like Chinese.
To "go deeper," the AI Challenger project provides datasets that are not only larger but also richer in semantic detail, allowing models to learn the relationships between objects and their properties in a way that mirrors human learning.
Methodology and Annotation
The creation of AIC involved a massive data collection and annotation effort, totaling over 185,000 worker hours. The raw images were crawled from the internet and filtered for quality and relevance.
- Human Keypoint Detection (HKD): The team annotated 14 skeletal keypoints (e.g., shoulders, elbows, knees) for 700,000 human instances. Each point includes a visibility flag: visible, labeled but not visible (occluded), or not labeled.
- Large-scale Attribute Dataset (LAD): They defined 240 classes across five subsets (animals, fruits, vehicles, electronics, and hairstyles) and annotated them with 359 different attributes.
- Image Chinese Captioning (ICC): For 300,000 images, they collected five unique Chinese captions from native speakers, encouraging the use of idioms and descriptive adjectives to ensure linguistic richness.
Sub-dataset Characteristics
- AIC-HKD: 300,000 images with 700,000 humans and 14 keypoints each. This is significantly larger than MSCOCO (200k images) or MPII (25k images).
- AIC-LAD: 81,658 images across 240 classes with 359 attributes. It includes both low-level visual attributes (color, shape) and high-level semantic ones (habits, functions).
- AIC-ICC: 300,000 images with 1.5 million Chinese captions. This is the first large-scale dataset of its kind for the Chinese language.
Notably, there is a 95% overlap between the images in the HKD and ICC datasets, providing a unique opportunity for researchers to explore multi-task learning.
Key Insights and Pose Diversity
Analysis of the dataset revealed several critical insights. First, semantic attributes matter; unlike previous datasets that focused on simple visual traits, AIC-LAD includes "human-concerned" semantic attributes, such as an animal's diet or an electronic device's function. This helps models learn concepts through description rather than just raw visual patterns.
Second, annotation difficulty varies by body part. Humans label upper body parts (shoulders, neck) more accurately than lower body parts (hips, ankles), which are often occluded. Finally, by aligning human figures from the dataset, the authors demonstrate a vast diversity of poses, from standard standing and sitting to complex athletic maneuvers.
Baseline Results
The authors established baselines for each task to prove the dataset's utility:
- HKD: Using a top-down approach, they achieved a mean Average Precision (mAP-14) of 0.234.
- LAD: In zero-shot recognition tests, the Metric-based Data Partitioning (MDP) method performed best, with an average accuracy of 48.58%.
- ICC: Their baseline "Show and Tell" model (adapted for Chinese) achieved a BLEU-4 score of 0.461 and a CIDEr score of 1.425. Qualitatively, models trained on ICC provided much more descriptive and context-aware captions compared to those trained on English datasets and translated.
Conclusion
The AI Challenger dataset provides a robust new benchmark for the global computer vision community. Future work may include adding facial expressions, specific hand actions, or expanding the captioning breadth to non-human scenes. The high overlap between sub-datasets remains a goldmine for future research into cross-task correlation.
Summary updated