Bugs in the Data: How ImageNet Misrepresents Biodiversity

Alexandra Sasha Luccioni, David Rolnick

Abstract

ImageNet-1k, the gold standard for benchmarking computer vision, contains significant flaws in its wildlife categories, which make up over a quarter of the dataset. An audit by expert ecologists reveals that 12.3% of wildlife images are mislabeled, class definitions frequently overlap, and the data suffers from a severe "Americentric" geographic bias that misrepresents global biodiversity.

Introduction

For over a decade, ImageNet has been the bedrock of computer vision research. While researchers have recently scrutinized the dataset for biases regarding people and man-made objects, the "wildlife" portion—comprising 27% of ImageNet-1k—has largely escaped expert review.

The authors argue that because ImageNet was curated by non-experts using web-scraped data and the WordNet hierarchy, it contains fundamental errors in how the natural world is categorized. If the "ground truth" labels used to evaluate our most advanced models are wrong, then our metrics for success are misleading. Furthermore, models trained on this data may inherit cultural and geographic biases, making them less effective or even harmful when deployed in global contexts like conservation or automated species identification.

Methodology

The researchers performed a systematic audit of the ImageNet-1k validation set, specifically focusing on the 269 classes representing wild, extant animals (13,450 images total). To ensure scientific accuracy, they recruited 20 expert annotators (ecologists and taxonomists) to evaluate the data.

The experts assessed class validity (overlapping definitions), image accuracy (correct labeling), image quality (blurriness or "fake" animals), and geographic origin. For bird classes, experts identified images down to the species level to quantify geographic bias.

Key Findings

The Expert Gap and Semantic Confusion

Many errors stem from the original annotators (Mechanical Turk workers) lacking domain knowledge. For example, the "centipede" class is 60% incorrect because non-experts frequently confuse them with millipedes or caterpillars. Furthermore, the dataset violates the core machine learning assumption of mutually exclusive classes; 11.9% of wildlife classes overlap. For instance, "meerkat" is a specific class, but meerkats are also "mongooses," which is a separate, broader class in the same dataset.

Contextual and Quality Bias

The dataset often captures animals in human-centric contexts rather than their natural habitats. Nearly 25% of fish images contain humans (often holding the fish as a trophy), and many "wild" animals are actually photographed in zoos or as pets. Additionally, 1.4% of the images feature "fake" animals like plushies or statues.

Contextual bias: A human holding a fish, a common occurrence in ImageNet's 'wildlife' categories.

Geographic Imperialism

ImageNet presents a Western-centric view of nature. Even for classes that exist globally, the images are overwhelmingly of North American or European species. If one searches for "Jay," ImageNet provides the Blue Jay (U.S./Canada) 62% of the time, despite there being dozens of other Jay species worldwide. While only 8.4% of the world's bird species are found in the U.S., the U.S. accounts for the vast majority of bird images in ImageNet. Diversity from Africa, Asia, and South America is severely under-represented.

Problematic content: A plush toy representing a wild animal, illustrating the 'fake' animals found in the dataset.

Quantitative Results

  • 12.3% Error Rate: Overall, more than 1 in 10 wildlife images are incorrectly labeled.
  • Taxonomic Disparity: Reptiles had a 27% error rate, while birds were the most accurate at 7%.
  • The "Ferret" Failure: The "black-footed ferret" class (an endangered wild species) was 98% incorrect, consisting almost entirely of images of domestic pet ferrets.
  • Problematic Content: 6.7% of images contain humans, 4.5% contain multiple species, and 1.4% feature "fake" animals.

Conclusion

The authors acknowledge that auditing the full training set of over 1.2 million images would require a prohibitive amount of expert labor. However, they suggest that the ML community must move away from using ImageNet as a "one-size-fits-all" benchmark. Future dataset curation, especially for complex real-world systems like biodiversity, must involve domain experts to avoid the "bugs" that currently skew reported model accuracy.

Summary updated

Abstract

ImageNet-1k is a dataset often used for benchmarking machine learning (ML) models and evaluating tasks such as image recognition and object detection. Wild animals make up 27% of ImageNet-1k but, unlike classes representing people and objects, these data have not been closely scrutinized. In the current paper, we analyze the 13,450 images from 269 classes that represent wild animals in the ImageNet-1k validation set, with the participation of expert ecologists. We find that many of the classes are ill-defined or overlapping, and that 12% of the images are incorrectly labeled, with some classes having >90% of images incorrect. We also find that both the wildlife-related labels and images included in ImageNet-1k present significant geographical and cultural biases, as well as ambiguities such as artificial animals, multiple species in the same image, or the presence of humans. Our findings highlight serious issues with the extensive use of this dataset for evaluating ML systems, the use of such algorithms in wildlife-related tasks, and more broadly the ways in which ML datasets are commonly created and curated.
BibTeX
@inproceedings{luccioni2023bugs,
  author = {Alexandra Sasha Luccioni and David Rolnick},
  title = {{Bugs in the Data: How ImageNet Misrepresents Biodiversity}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26682},
}
Figure