Detecting Mammals in UAV Images: Best Practices to address a substantially Imbalanced Dataset with Deep Learning

Benjamin Kellenberger, Diego Marcos, Devis Tuia

TL;DR

This paper provides a roadmap for using Deep Learning to detect mammals in large-scale UAV imagery, a task plagued by extreme class imbalance where animals are "needles in a haystack." By combining curriculum learning, hard negative mining, and a novel "border class," the authors reduced false positives by an order of magnitude, cutting the manual verification workload for park rangers by two-thirds while maintaining a 90% animal detection rate.

What Problem Does This Paper Address?

Wildlife conservation relies on accurate animal censuses, but traditional methods—like ground surveys or manned aircraft flights—are expensive, dangerous, and limited in scope. While Unmanned Aerial Vehicles (UAVs) offer a safer, cheaper alternative, they generate a staggering amount of data.

The technical challenge is twofold:

  1. Extreme Class Imbalance: In a typical wildlife reserve, the vast majority of aerial images contain only grass, rocks, or trees. Animals are incredibly rare, meaning a standard model might simply learn to predict "background" every time to achieve high accuracy.
  2. Background Heterogeneity: Objects like dead tree trunks or dirt mounds frequently mimic the appearance of animals, leading to thousands of false alarms that overwhelm human operators who must verify the detections.
Figure 3: Full extents of a sample image. The single animal present in this scene (enlarged in the red bounding box) is difficult to locate, both for humans and machines.

Methods

The authors deployed a Convolutional Neural Network (CNN) based on a ResNet-18 architecture, fine-tuned specifically for animal detection. Their core contribution lies in a suite of training "best practices" designed to handle imbalanced data:

  • Curriculum Learning: To prevent the model from being "drowned" by background examples early on, they initially trained only on patches containing animals. They then gradually introduced the full, background-heavy dataset.
  • The Border Class: Standard detectors often struggle with "spillage," where the model's receptive field sees part of an animal but is centered on the background. The authors introduced a dedicated "border" class for pixels immediately surrounding an animal, helping the model distinguish between a true animal center and its periphery.
  • Hard Negative Mining: The model was periodically tasked with finding its own worst mistakes (high-confidence false positives). These "hard negatives" were then given extra weight in subsequent training iterations to force the model to learn subtle differences between animals and look-alikes.
  • Delayed Rotational Augmentation: While rotating images helps the model generalize, the authors found that doing this too early confused the network. They achieved better results by introducing rotations only during the final fine-tuning stage.

Datasets

The study utilized a dataset acquired over the Kuzikus wildlife reserve in Namibia.

  • Scale: 654 high-resolution RGB images (3000 x 4000 pixels) covering 13.38 $km^2$.
  • Content: 1,183 annotated large mammals (including Kudus, Gemsboks, and Rhinos).
  • Imbalance: As shown in the distribution below, the vast majority of images contain zero animals, highlighting the difficulty of the task.
Figure 2: Distribution of the number of animals per image for the training, validation and test sets. Standard deviation ranges are given for the three cross-validation splits. The majority of images does not contain animals, which poses a significant challenge to any detector.

Key Insights and Ideas

  • Precision at High Recall is the Goal: In a census, missing an animal (a false negative) is often worse than a false alarm. However, if the false alarm rate is too high, the system is useless. The authors focused on maximizing precision specifically at the 80-90% recall mark.
  • Contextual Confusion: The paper highlights how easily nature tricks AI. Dead trees can look like ungulates, and shadows of dirt mounds can look like ostriches.
  • Verification-Centric Evaluation: The authors argue that standard computer vision metrics (like Intersection-over-Union) are too harsh for wildlife counting. They proposed a "census-oriented" protocol that counts a detection as correct if it falls within a reasonable distance of the animal, prioritizing the count over pixel-perfect localization.
Figure 4: Examples of animal-to-background confusion: dead tree trunks (4a) can easily be mistaken for ungulates (4b); shadows of dirt mounds (4c) may look like ostriches (4d).

Key Results

The proposed "Full Model" significantly outperformed the previous state-of-the-art baseline:

  • False Positive Reduction: At an 80% recall rate, the CNN produced roughly 447 false positives, compared to over 2,500 from the baseline model.
  • Efficiency for Rangers: To find 90% of the animals, a ranger using the baseline model would have to screen almost every image tile. The proposed CNN reduced the number of tiles requiring manual verification by three times.
  • Spatial Accuracy: The CNN was found to be much more precise in its positioning, often detecting the animal itself rather than just its shadow.

Limitations and Future Work

  • Image Overlap: The current system does not account for the fact that one animal might appear in multiple overlapping UAV images, which could lead to double-counting in a final census.
  • Environment Specificity: While the training methods are model-agnostic, the current model is tuned for the African savanna. Adapting it to dense forests or different climates would require new training data.
  • Automation Gap: While the workload is reduced, a human "in the loop" is still required to reach 100% accuracy. Future work could explore even more advanced augmentation, such as using Generative Adversarial Networks (GANs) to create synthetic training examples of rare species.

Summary updated

Abstract

Knowledge over the number of animals in large wildlife reserves is a vital necessity for park rangers in their efforts to protect endangered species. Manual animal censuses are dangerous and expensive, hence Unmanned Aerial Vehicles (UAVs) with consumer level digital cameras are becoming a popular alternative tool to estimate livestock. Several works have been proposed that semi-automatically process UAV images to detect animals, of which some employ Convolutional Neural Networks (CNNs), a recent family of deep learning algorithms that proved very effective in object detection in large datasets from computer vision. However, the majority of works related to wildlife focuses only on small datasets (typically subsets of UAV campaigns), which might be detrimental when presented with the sheer scale of real study areas for large mammal census. Methods may yield thousands of false alarms in such cases. In this paper, we study how to scale CNNs to large wildlife census tasks and present a number of recommendations to train a CNN on a large UAV dataset. We further introduce novel evaluation protocols that are tailored to censuses and model suitability for subsequent human verification of detections. Using our recommendations, we are able to train a CNN reducing the number of false positives by an order of magnitude compared to previous state-of-the-art. Setting the requirements at 90% recall, our CNN allows to reduce the amount of data required for manual verification by three times, thus making it possible for rangers to screen all the data acquired efficiently and to detect almost all animals in the reserve automatically.
BibTeX
@inproceedings{kellenberger2018detecting,
  author = {Benjamin Kellenberger and Diego Marcos and Devis Tuia},
  title = {{Detecting Mammals in UAV Images: Best Practices to address a substantially Imbalanced Dataset with Deep Learning}},
  booktitle = {Remote},
  year = {2018},
  doi = {10.1016/j.rse.2018.06.028},
}
Figure