Deep learning-based ecological analysis of camera trap images is impacted by training data quality and size

Omiros Pantazis, Peggy Bevan, Holly Pringle, Guilherme Braga Ferreira, Daniel J. Ingram, Emily Madsen, Liam Thomas, Dol Raj Thanet, Thakur Silwal, Santosh Rayamajhi, Gabriel J. Brostow, Oisin Mac Aodha, Kate E. JonesarXiv 2024 · arXiv · OpenReview

TL;DR

Deep learning models can accurately estimate ecological metrics like species richness from camera trap data, even with imperfect training data. However, the quality and size of the training data significantly impact species-specific metrics, while model architecture has minimal effect. This suggests prioritizing clean, large datasets over complex models for reliable ecological insights.

What Problem Does This Paper Address?

Biodiversity monitoring relies heavily on camera trap data, which provides valuable insights into species richness, occupancy, and activity patterns. However, manually processing the vast number of images generated is a major bottleneck, leading to significant delays in obtaining ecological information. Deep learning (DL) offers a promising solution for automating this process through image classification. Despite the widespread adoption of DL in this field, there's a critical knowledge gap regarding how decisions made during the DL model training process—such as the choice of model architecture, the quality of the training data (e.g., presence of mislabeled images), and the size of the training dataset—actually influence the accuracy of the downstream ecological metrics derived from these classifications. This paper aims to bridge this gap by systematically evaluating these impacts.

Methods

The researchers employed a multi-faceted approach to investigate the impact of DL training parameters on ecological metrics. They utilized two primary deep neural network architectures: ResNet (with varying depths: 18, 50, and 101) and ConvNeXt-T. Transfer learning, using weights pre-trained on ImageNet, was applied to all models. To simulate varying training data quality and quantity, they manipulated:

  • Model Architecture: Comparing the performance of different ResNet depths and ConvNeXt-T.
  • Training Set Label Noise: Creating six versions of the training data with label error rates ranging from 1% to 50%.
  • Training Set Size: Generating seven versions of the training data, reducing the number of labels from 100% down to 1%.

Ecological metrics were then calculated using both expert-labeled data and the DL-predicted labels from these manipulated datasets. These metrics included:

  • Species Richness: Count of wild species observed per camera trap location.
  • Occupancy: Modeled using a multi-species occupancy framework, accounting for imperfect detection.
  • Activity Patterns: Estimated using circular kernel density functions, with overlap coefficients used for comparison.

To quantify the relationship between DL performance and ecological accuracy, they calculated correlation metrics.

Datasets

The study utilized two distinct camera trap datasets:

  • Maasai Mara Camera Traps (MMCT): Collected from an African savannah in Kenya, comprising 82,300 labeled images and 47 species.
  • Bardia Camera Traps (BCT): Collected from an Asian sub-tropical dry forest in Nepal, comprising 40,308 labeled images and 29 species.

These datasets were chosen for their ecological diversity and differences in size and species composition, aiming to enhance the generalizability of the findings.

Key Insights and Ideas

  • Model Architecture is Less Critical Than Data Quality: Surprisingly, the choice of DL model architecture (e.g., ResNet vs. ConvNeXt-T, or different ResNet depths) had a minimal impact on the accuracy of the derived ecological metrics. This suggests that practitioners might not need to chase the latest, most complex architectures if their data is well-curated.
  • Robustness to Noise and Size Reduction: Ecological metrics, particularly species richness, proved remarkably resilient to imperfections in the training data. They could tolerate up to 10% label noise and a 50% reduction in training set size without significant degradation in accuracy. This is a crucial finding for real-world applications where perfect data is rare.
  • Species-Specific Metrics are More Sensitive: While community-level metrics like species richness were robust, species-specific metrics (e.g., occupancy for rarer or visually similar species) were disproportionately affected by noise and reduced dataset size. This highlights the importance of careful data curation for detailed ecological analyses.
  • Conventional ML Metrics Can Be Misleading: The study found that standard machine learning evaluation metrics (like classification accuracy) don't always correlate well with the accuracy of the ecological metrics. A model with high classification accuracy might not necessarily produce accurate ecological insights, and vice-versa. This emphasizes the need to evaluate DL models based on their performance on the actual ecological tasks they are intended for.
  • Class Imbalance is a Major Hurdle: The paper implicitly and explicitly points to class imbalance (i.e., some species being much rarer than others) as a significant challenge that disproportionately affects the detection and accurate classification of less common species.

Key Results

  • Species Richness: DL-predicted species richness closely matched expert-labeled data and was robust to model architecture variations. It was impacted by label noise above 10% and reduced training set size, particularly in the smaller BCT dataset.
  • Occupancy: DL models could recover some species-specific responses to environmental covariates, even with significant data manipulation. However, less common and visually similar species were less consistently predicted. For example, elephant and zebra responses were consistent across manipulations, while eland and topi were not. Grant's gazelle and Thomson's gazelle detections were conflated.
  • Activity Patterns: DL models produced activity patterns with high overlap coefficients (0.8-1) with expert-labeled data. Accuracy was robust to noise up to 10% and a 50% drop in training set size. Model manipulations primarily led to an overestimation of diurnal predictions.
  • Correlation: Ecological metric accuracy showed varying correlations with classification error. Activity pattern accuracy was more strongly correlated with DL performance than occupancy accuracy.

Limitations and Future Work

  • Uniform Label Noise Simulation: The study simulated label noise uniformly across species. The authors acknowledge that real-world label error might be more focused on specific species or image types (e.g., nocturnal images), which could lead to different outcomes. Future work could explore more realistic noise distributions.
  • Temporal vs. Spatial Data Split: A temporal data split was used for model training and testing. The authors note that a spatial split might be more conventional and could potentially lead to a greater impact of reduced training set size on generalization to new sites.
  • Focus on Specific Metrics: The study focused on species richness, occupancy, and activity patterns. Future research could explore the impact of training data quality on other ecological metrics.
  • Enhancing Accessibility: The authors suggest that future research should focus on enhancing the accessibility of DL tools, ensuring they can be widely adopted by the global conservation community, especially for practitioners with limited access to high-performance computing. This could involve developing more efficient methods for data annotation and model training.

Summary updated

Abstract

Large wildlife image collections from camera traps are crucial for biodiversity monitoring, offering insights into species richness, occupancy, and activity patterns. However, manual processing of these data is time-consuming, hindering analytical processes. To address this, deep neural networks have been widely adopted to automate image analysis. Despite their growing use, the impact of model training decisions on downstream ecological metrics remains unclear. Here, we analyse camera trap data from an African savannah and an Asian sub-tropical dry forest to compare key ecological metrics derived from expert-generated species identifications with those generated from deep neural networks. We assess the impact of model architecture, training data noise, and dataset size on ecological metrics, including species richness, occupancy, and activity patterns. Our results show that while model architecture has minimal impact, large amounts of noise and reduced dataset size significantly affect these metrics. Nonetheless, estimated ecological metrics are resilient to considerable noise, tolerating up to 10% error in species labels and a 50% reduction in training set size without changing significantly. We also highlight that conventional metrics like classification error may not always be representative of a model's ability to accurately measure ecological metrics. We conclude that ecological metrics derived from deep neural network predictions closely match those calculated from expert labels and remain robust to variations in the factors explored. However, training decisions for deep neural networks can impact downstream ecological analysis. Therefore, practitioners should prioritize creating large, clean training sets and evaluate deep neural network solutions based on their ability to measure the ecological metrics of interest.
BibTeX
@misc{pantazis2024deep,
  author = {Omiros Pantazis and Peggy Bevan and Holly Pringle and Guilherme Braga Ferreira and Daniel J. Ingram and Emily Madsen and Liam Thomas and Dol Raj Thanet and Thakur Silwal and Santosh Rayamajhi and Gabriel J. Brostow and Oisin Mac Aodha and Kate E. Jones},
  title = {{Deep learning-based ecological analysis of camera trap images is impacted by training data quality and size}},
  year = {2024},
  eprint = {2408.14348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.14348},
}
Figure