Image identification of Protea species with attributes and subgenus scaling

Peter Thompson, Willie BrinkWACV 2020

TL;DR

This paper tackles the challenging problem of automatically identifying Protea species from images, a crucial task for biodiversity monitoring. They introduce a novel three-part model that combines a CNN with attention, subgenus classification for score rescaling, and attribute-based probabilistic modeling, achieving significant improvements over baseline methods.

What Problem Does This Paper Address?

The Cape Floristic Region in South Africa is a biodiversity hotspot, and the Protea genus is a key indicator of its health. Monitoring these plants is vital for conservation, but traditional methods relying on expert scientists are becoming overwhelmed by the influx of data from crowd-sourced platforms like iNaturalist. While these platforms offer vast amounts of image data, automatic species identification faces several hurdles:

  • Fine-grained classification: Many Protea species look remarkably similar, making visual distinction difficult.
  • Data scarcity: Rarer species have very few images available for training.
  • Class imbalance: A few common species dominate the datasets, while many others are underrepresented.
  • Variability: Images are taken by diverse observers, leading to variations in quality, composition, and background.

This paper aims to develop a robust automated system to address these challenges, using Protea identification as a proxy for fine-grained, imbalanced citizen science datasets.

Methods

The authors propose a three-component model:

  1. CNN with Attention (CNN-A): This component uses a pre-trained Inception-V3 network with a multi-region attention mechanism. The attention mechanism is designed to focus on the flowerhead, which is the most discriminative part of the plant for identification. They extract two attention regions, with the primary one focusing on the inflorescence. This attention-boosted image is then used to train a CNN for species classification.
  2. Subgenus Scaling: Recognizing that Protea species can be grouped into more distinct subgenera, this component uses a separate CNN trained on these coarser categories. The output of this subgenus classifier is used to rescale the class scores from the species-level CNN-A, effectively adjusting the confidence scores based on broader taxonomic relationships.
  3. Attribute-based Probabilistic Model: This part incorporates optional but valuable contextual information associated with each observation: location, elevation, and date. A Bayesian approach is used, assuming conditional independence between these attributes, to further refine the species identification probabilities. This leverages the fact that Protea species often have specific geographical and temporal distributions.

Datasets

  • iNaturalist Dataset: The authors collected a new dataset from iNaturalist, focusing on "research-grade" observations of Protea species within the Cape Floristic Region. They filtered for images of flowering inflorescences and restricted the dataset to 41 species for which at least 20 images were available. This resulted in 4,849 images.
  • Protea Atlas Project: Data from the Protea Atlas Project was used to construct conditional probability distributions for species based on location, elevation, and flowering time. This information was used to inform the attribute-based component of the model.

Key Insights and Ideas

  • Attention is crucial for fine-grained tasks: The multi-region attention mechanism significantly improves performance by helping the model focus on the most informative parts of the image, especially when dealing with visually similar species.
  • Hierarchical classification aids fine-tuning: Using subgenus classification as an intermediate step to rescale species-level scores provides a performance boost, likely by leveraging more robust, less imbalanced, and more distinct intermediate categories.
  • Contextual attributes are powerful: Incorporating location, elevation, and date information, even with a simple Bayesian model, provides a substantial improvement in identification accuracy, highlighting the importance of ecological context.
  • Crowd-sourced data is a double-edged sword: While offering vast quantities of images, it also introduces challenges like imbalance and variability that require sophisticated modeling techniques.

Key Results

The paper presents a comprehensive ablation study comparing various configurations of their model. The full model, combining CNN-A with attention, subgenus scaling, and all attributes (location, elevation, date), achieved the best performance.

  • Top-1 Accuracy: The full model achieved a top-1 accuracy of 70.43%.
  • Recall: The full model achieved a recall of 66.88%.
  • Comparison: This significantly outperforms baselines like a random classifier (2.44% recall) and a standard CNN without attention (13.51% recall). Even the CNN-A with attention alone achieved 35.59% recall, demonstrating the value of attention. Incorporating attributes with a subgenus prior also showed substantial gains.

The per-class accuracy graph (Figure 6) indicates that the model performs relatively consistently across species, even those with fewer training images, suggesting it mitigates the class imbalance issue effectively.

Limitations and Future Work

  • Dataset Size: While the dataset is new and valuable, the authors acknowledge that for many rarer species, the number of available images is still limited.
  • Attribute Dependence: The performance gain from attributes is dependent on their availability. In cases where attributes are missing, the model relies more heavily on the image-based components.
  • Future Work: The authors suggest that their model can serve as a foundation for future research. Potential avenues include exploring more sophisticated attribute integration methods, investigating active learning strategies to efficiently expand the dataset, and applying this approach to other fine-grained classification problems in biodiversity monitoring.

Summary updated

BibTeX
@inproceedings{thompson2020image,
  author = {Peter Thompson and Willie Brink},
  title = {{Image identification of Protea species with attributes and subgenus scaling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure