Abstract
ImageNet-1k, the gold standard for benchmarking computer vision, contains significant flaws in its wildlife categories, which make up over a quarter of the dataset. An audit by expert ecologists reveals that 12.3% of wildlife images are mislabeled, class definitions frequently overlap, and the data suffers from a severe "Americentric" geographic bias that misrepresents global biodiversity.
Introduction
For over a decade, ImageNet has been the bedrock of computer vision research. While researchers have recently scrutinized the dataset for biases regarding people and man-made objects, the "wildlife" portion—comprising 27% of ImageNet-1k—has largely escaped expert review.
The authors argue that because ImageNet was curated by non-experts using web-scraped data and the WordNet hierarchy, it contains fundamental errors in how the natural world is categorized. If the "ground truth" labels used to evaluate our most advanced models are wrong, then our metrics for success are misleading. Furthermore, models trained on this data may inherit cultural and geographic biases, making them less effective or even harmful when deployed in global contexts like conservation or automated species identification.
Methodology
The researchers performed a systematic audit of the ImageNet-1k validation set, specifically focusing on the 269 classes representing wild, extant animals (13,450 images total). To ensure scientific accuracy, they recruited 20 expert annotators (ecologists and taxonomists) to evaluate the data.
The experts assessed class validity (overlapping definitions), image accuracy (correct labeling), image quality (blurriness or "fake" animals), and geographic origin. For bird classes, experts identified images down to the species level to quantify geographic bias.
Key Findings
The Expert Gap and Semantic Confusion
Many errors stem from the original annotators (Mechanical Turk workers) lacking domain knowledge. For example, the "centipede" class is 60% incorrect because non-experts frequently confuse them with millipedes or caterpillars. Furthermore, the dataset violates the core machine learning assumption of mutually exclusive classes; 11.9% of wildlife classes overlap. For instance, "meerkat" is a specific class, but meerkats are also "mongooses," which is a separate, broader class in the same dataset.
Contextual and Quality Bias
The dataset often captures animals in human-centric contexts rather than their natural habitats. Nearly 25% of fish images contain humans (often holding the fish as a trophy), and many "wild" animals are actually photographed in zoos or as pets. Additionally, 1.4% of the images feature "fake" animals like plushies or statues.
Geographic Imperialism
ImageNet presents a Western-centric view of nature. Even for classes that exist globally, the images are overwhelmingly of North American or European species. If one searches for "Jay," ImageNet provides the Blue Jay (U.S./Canada) 62% of the time, despite there being dozens of other Jay species worldwide. While only 8.4% of the world's bird species are found in the U.S., the U.S. accounts for the vast majority of bird images in ImageNet. Diversity from Africa, Asia, and South America is severely under-represented.
Quantitative Results
- 12.3% Error Rate: Overall, more than 1 in 10 wildlife images are incorrectly labeled.
- Taxonomic Disparity: Reptiles had a 27% error rate, while birds were the most accurate at 7%.
- The "Ferret" Failure: The "black-footed ferret" class (an endangered wild species) was 98% incorrect, consisting almost entirely of images of domestic pet ferrets.
- Problematic Content: 6.7% of images contain humans, 4.5% contain multiple species, and 1.4% feature "fake" animals.
Conclusion
The authors acknowledge that auditing the full training set of over 1.2 million images would require a prohibitive amount of expert labor. However, they suggest that the ML community must move away from using ImageNet as a "one-size-fits-all" benchmark. Future dataset curation, especially for complex real-world systems like biodiversity, must involve domain experts to avoid the "bugs" that currently skew reported model accuracy.
Summary updated