Intelligent Image Collection: Building the Optimal Dataset

Matthew Gwilliam, Ryan FarrellWACV 2020 · OpenReview

Abstract

Key recognition tasks such as fine-grained visual categorization (FGVC) have benefited from increasing attention among computer vision researchers. The development and evaluation of new approaches relies heavily on benchmark datasets; such datasets are generally built primarily with categories that have images readily available, omitting categories with insufficient data. This paper takes a step back and rethinks dataset construction, focusing on intelligent image collection driven by: (i) the inclusion of all desired categories, and, (ii) the recognition performance on those categories. Based on a small, author-provided initial dataset, the proposed system recommends which categories the authors should prioritize collecting additional images for, with the intent of optimizing overall categorization accuracy. We show that mock datasets built using this method outperform datasets built without such a guiding framework. Additional experiments give prospective dataset creators intuition into how, based on their circumstances and goals, a dataset should be constructed.

Introduction: The Problem of "Low-Hanging Fruit"

In the field of Fine-Grained Visual Categorization (FGVC)—distinguishing between very similar sub-categories like specific bird species or car models—performance is usually limited by the quality and distribution of the dataset.

Most datasets are currently built using "low-hanging fruit": images easily scraped from the web. This leads to massive geographic and category biases. For example, the iNaturalist dataset has an abundance of bird data for the US but almost none for the hyper-biodiverse Amazon Rainforest. Building a representative dataset for the Amazon requires expensive, time-consuming expeditions. If a researcher can only afford to collect a limited number of additional images, standard practice is to collect whatever is available. This paper argues we should instead target the "hard" classes that the model currently struggles to distinguish.

Geographic bias in existing datasets: High data density in the US vs. sparse data in the Amazon.

Methodology: The Intelligent Collection Loop

The authors propose a systematic framework to guide dataset expansion through an iterative loop:

  1. Base Dataset: Start with a small, balanced set of images for all desired categories (the "base" number).
  2. Difficulty Assessment: Train a model (like a ResNet-50) using 5-fold cross-validation on this base set.
  3. Ranking: Evaluate the model's performance on each class using various metrics: per-class accuracy, loss, precision, F1-score, or KL-Divergence (to measure visual similarity/confusion).
  4. Targeted Collection: Identify the bottom 50% of classes (the "worst" performers) and prioritize them for the next round of image collection (the "jump" size).

The researchers tested this using "Mock Datasets." They took existing large datasets (CUB-200-2011, Stanford Cars, Stanford Dogs, FGVC-Aircraft, and NABirds), hid most of the data in a "bank," and treated a small subset as the starting point to simulate the real-world process of deciding what to collect next.

The framework prioritizes adding images to visually similar "hard" classes to maximize accuracy gains.

Key Results and Insights

  • Intelligent vs. Random: In almost every scenario, "Intelligent" collection outperformed random collection. For example, on the Cars dataset, the intelligent approach reached 78.43% accuracy compared to 76.82% for random collection.
  • The "Hard" Class Priority: Adding images to categories with high visual similarity (e.g., two types of terns) yields much higher accuracy gains than adding images to easily distinguishable categories (e.g., a hummingbird vs. a hawk).
  • Robustness to Architecture: The "hard" classes for one model (e.g., ResNet) are generally the same for another (e.g., DenseNet). This means you can use a cheap model to decide what data to collect for a more expensive one later.
  • The "Octile" Effect: Granular analysis showed that adding images to the "hardest" 1/8th of classes (Octile 1) resulted in a 34.75% average accuracy improvement for those specific classes, whereas adding to the "easiest" classes (Octile 8) only yielded a 9.80% improvement.

Conclusion and Future Work

This framework challenges the "more is always better" approach to data collection. By using a small initial dataset to identify which categories a model finds most difficult, researchers can prioritize specific images that maximize overall classification accuracy, outperforming traditional random collection methods.

Future work will focus on "Adaptive Jumps" (adding images proportional to difficulty) and integrating the financial cost of collecting different species. While the current method requires at least some images of every class to start, it provides a robust foundation for building more efficient and accurate computer vision benchmarks.

Summary updated

Abstract

Key recognition tasks such as fine-grained visual categorization (FGVC) have benefited from increasing attention among computer vision researchers. The development and evaluation of new approaches relies heavily on benchmark datasets; such datasets are generally built primarily with categories that have images readily available, omitting categories with insufficient data. This paper takes a step back and rethinks dataset construction, focusing on intelligent image collection driven by: (i) the inclusion of all desired categories, and, (ii) the recognition performance on those categories. Based on a small, author-provided initial dataset, the proposed system recommends which categories the authors should prioritize collecting additional images for, with the intent of optimizing overall categorization accuracy. We show that mock datasets built using this method outperform datasets built without such a guiding framework. Additional experiments give prospective dataset creators intuition into how, based on their circumstances and goals, a dataset should be constructed.
BibTeX
@inproceedings{gwilliam2020intelligent,
  author = {Matthew Gwilliam and Ryan Farrell},
  title = {{Intelligent Image Collection: Building the Optimal Dataset}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=i478XwVDYnc},
}
Figure