DivShift: Exploring Domain-Specific Distribution Shifts in Large-Scale, Volunteer-Collected Biodiversity Datasets

Elena Sierra, Lauren E. Gillespie, Salim Soltani, Moisés Expósito-Alonso, Teja KattenbornAAAI 2025 · DOI

Abstract

Large-scale biodiversity datasets, such as those from iNaturalist, are increasingly used for fine-grained visual classification. However, these datasets are "opportunistic," collected by volunteers with inherent biases. The DivShift framework provides a systematic approach to quantify how these real-world biases—spatial, temporal, taxonomic, observer, and sociopolitical—impact machine learning model performance. Using a new dataset of 7.5 million images, we demonstrate that while models show some resilience, specific shifts like observer engagement and human geographic footprints significantly degrade accuracy.

Introduction: The Challenge of Opportunistic Data

Citizen science has revolutionized biodiversity monitoring, providing a wealth of data for AI development. Yet, this data is non-randomly sampled. Volunteers tend to photograph species near urban centers (spatial bias), during weekends or specific events (temporal bias), and focus on "charismatic" species (taxonomic bias). The critical question is how these sampling strategies affect the reliability of AI models in real-world ecological applications. If a model is trained primarily on suburban observations, its performance on rare species in remote wilderness remains uncertain.

Methodology: The DivShift Framework

The DivShift framework treats domain-specific biases as distribution shifts. The process follows three main steps:

  1. Partitioning: The dataset is split into subsets ($P_A$ and $P_B$) based on expert-verified biases, such as "Wilderness" vs. "Human-Modified" land.
  2. Quantifying Shift: The Jensen-Shannon Distance (JSD) is employed to measure the divergence in label distributions between these partitions.
  3. Performance Evaluation: Models (e.g., ResNet, ViT) are trained on one partition and tested on others to measure the "performance gap."
Quantifying distribution shifts using label distributions and Jensen-Shannon Distance (JSD).

Partitions are categorized as strongly biased if the drop in accuracy exceeds the label shift, or weakly biased if the model generalizes better than the shift would suggest.

The DivShift-NAWC Dataset

To support this research, we introduce DivShift-NAWC (North American West Coast). This dataset contains nearly 7.5 million images covering over 7,500 plant species across seven terrestrial biomes. The data is partitioned across five primary bias types:

Overview of the five types of domain-specific biases explored in the DivShift framework.
  • Spatial: Wilderness vs. Human-Modified areas based on the Global Human Footprint Index.
  • Temporal: Observations during the "City Nature Challenge" vs. standard periods.
  • Taxonomic: Naturally long-tailed vs. artificially balanced species distributions.
  • Observer: "Engaged" power users (>1,000 observations) vs. "Casual" users (<50 observations).
  • Sociopolitical: Shifts across state and national borders (e.g., California vs. Sonora).

Key Insights and Results

Our analysis reveals several critical findings regarding model generalization:

The "Human Footprint" Asymmetry

Models trained on human-modified areas generalize well to wilderness. Conversely, wilderness-trained models struggle in modified environments (an 11.3% drop in Top-1 accuracy). This suggests that wilderness data may be too sparse to capture the full phenotypic diversity present in more populated regions.

Observer Engagement and Data Quality

The most dramatic performance gap involves observer engagement. Models trained on data from "engaged" observers provide a +4.9% boost when tested on casual data. However, models trained on "casual" data see a 23.1% accuracy drop when tested on engaged data, indicating that power users provide significantly higher-quality training signals.

The Impact of Sociopolitical Boundaries

Interestingly, sociopolitical boundaries (state lines) often create sharper distribution shifts than ecological boundaries. This highlights how human policy and regional data availability significantly shape the AI's perception of the natural world.

Model Scaling

While larger architectures like Vision Transformers (ViT) improve absolute accuracy by 15-20% over ResNet-18, the relative performance drops caused by distribution shifts remain consistent. This suggests that architectural improvements alone are insufficient to overcome fundamental data biases.

Limitations and Future Directions

While DivShift identifies performance drops, it does not yet explain the underlying causal mechanisms (e.g., lighting vs. morphology). Future work will explore self-supervised learning, implement per-observation data splitting to prevent leakage, and move toward measuring shifts in "environmental space" (e.g., climate hulls) to better capture ecological nuances.

Summary updated

BibTeX
@inproceedings{sierra2025divshift,
  author = {Elena Sierra and Lauren E. Gillespie and Salim Soltani and Moisés Expósito-Alonso and Teja Kattenborn},
  title = {{DivShift: Exploring Domain-Specific Distribution Shifts in Large-Scale, Volunteer-Collected Biodiversity Datasets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35060},
}

Loading related content...

Figure