Bugs in the Data: How ImageNet Misrepresents Biodiversity

Alexandra Sasha Luccioni, David Rolnick

Abstract

ImageNet-1k serves as the foundational benchmark for computer vision, yet its wildlife data—which constitutes over 27% of the dataset—has escaped rigorous scrutiny. This study presents a deep-dive audit conducted by 20 expert ecologists and taxonomists to evaluate the biological accuracy of ImageNet’s wildlife categories. Our findings reveal that 12.3% of these images are mislabeled, and many class definitions are scientifically unsound. Furthermore, the dataset exhibits a significant "Americentric" geographic bias and contains frequent "contextual poisoning" from human presence. These errors create an artificial ceiling for model performance and risk embedding cultural and geographic biases into models deployed for global conservation efforts.

Introduction

Machine learning (ML) progress is measured against benchmarks, with ImageNet-1k standing as the primary metric for image classification. While the "people" and "object" categories within ImageNet have been extensively audited for privacy concerns and social bias, the wildlife classes have largely been accepted as ground truth. However, because these labels were originally generated by non-experts using web-scraped data and the WordNet hierarchy, they often fail to reflect biological reality.

This paper argues that the errors in ImageNet’s wildlife data are not merely noise but systemic flaws. If the "ground truth" used to train and evaluate models is fundamentally incorrect, our understanding of ML progress in computer vision may be an illusion. This is particularly concerning for models intended for ecological monitoring or biodiversity conservation, where taxonomic precision is critical.

Methodology

To move beyond the limitations of crowdsourced labeling, the researchers recruited 20 expert annotators, including ecologists and taxonomists with graduate-level training. The study focused on 269 classes of "wild extant animals" within the ImageNet-1k validation set, totaling 13,450 images.

The experts conducted a two-tiered validation process:

  1. Class Validation: Assessing whether the category itself was clearly defined, represented a single species, or suffered from taxonomic overlap with other categories.
  2. Image Validation: Reviewing individual images for correct identification, the presence of humans, "fake" animals (such as toys or sculptures), image quality (blurriness), and the presence of multiple species.

Additionally, the team performed a species-level re-annotation of all 2,950 bird images in the dataset to specifically quantify geographic and taxonomic bias.

Taxonomic Chaos and Contextual Poisoning

The audit revealed significant structural issues in how ImageNet categorizes the natural world.

Taxonomic Overlap

Approximately 11.9% of the animal classes overlap. For example, "meerkat" and "mongoose" exist as distinct classes, yet meerkats are biologically a type of mongoose. This violates the fundamental machine learning assumption that classes should be mutually exclusive, leading to arbitrary penalties for models that correctly identify a meerkat as a mongoose.

The "Toy" Problem and Human Presence

Biological data is often "poisoned" by non-biological context. The audit found that 1.4% of wildlife images are actually stuffed animals, sculptures, or illustrations. Furthermore, 6.7% of images contain humans. A notable example is the "fish" categories, which frequently feature trophy photos of humans holding fish. This risks training models to associate the concept of a species with human hands or boats rather than the animal's natural habitat.

Geographic and Cultural Bias

The dataset reflects a heavy Western bias, often prioritizing cultural significance over biological diversity. This is most evident in the "Bald Eagle" effect: while there are over 60 species of eagles globally, ImageNet-1k primarily features the Bald Eagle, the national bird of the United States.

In the bird-specific re-annotation, 71% of images depicted species found in the U.S., despite the U.S. hosting only a small fraction of global avian diversity. This geographic skew limits the utility of models trained on ImageNet for global biodiversity applications, as they are effectively tuned to a North American lens.

Key Results

The expert audit produced several startling metrics regarding the quality of the ImageNet-1k validation set:

  • Overall Error Rate: 12.3% of wildlife images are incorrectly labeled.
  • Extreme Class Failures: Certain classes are almost entirely inaccurate. The "black-footed ferret" class, for instance, has a 98% error rate, with the vast majority of images actually depicting domestic ferrets.
  • Ambiguity: 4.5% of images contain multiple different species, creating significant confusion for standard classification tasks that expect a single label.

Discussion and Future Work

The findings suggest that simply "cleaning" the labels is insufficient. Many images are of such poor quality or depict "fake" animals that they must be replaced entirely to create a valid biological benchmark.

The authors propose several paths forward for the ML community:

  1. Expert-In-The-Loop Curation: Domain experts (biologists, taxonomists, and sociologists) should be involved at the start of dataset creation, not just during post-hoc audits.
  2. Structural Reform: The rigid WordNet hierarchy should be replaced with more flexible, scientifically accurate taxonomic structures that account for biological relationships.
  3. Impact Assessment: Further research is required to determine how these "bugs" in the data affect the performance of models when they are fine-tuned for real-world ecological applications, such as camera trap monitoring or species identification apps.

By addressing these systemic flaws, the computer vision community can move toward benchmarks that truly reflect the complexity and diversity of the natural world.

Summary updated

BibTeX
@inproceedings{luccioni2023bugs,
  author = {Alexandra Sasha Luccioni and David Rolnick},
  title = {{Bugs in the Data: How ImageNet Misrepresents Biodiversity}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i12.26682},
}
Figure