MinneApple: A Benchmark Dataset for Apple Detection and Segmentation

Nicolai Häni, Pravakar Roy, Volkan Isler

TL;DR

MinneApple is a massive new open-source dataset featuring over 41,000 annotated apples across 1,000 high-resolution orchard images. By providing precise polygonal masks and a standardized benchmark, the authors aim to move agricultural robotics away from "ad-hoc" private datasets toward robust, comparable computer vision solutions for fruit detection and yield estimation.

What Problem Does This Paper Address?

Automating fruit harvesting and yield mapping is a "holy grail" for precision agriculture, but the "wild" environment of an orchard is a computer vision nightmare. Unlike the clean, centered images found in general datasets like COCO or ImageNet, orchard images suffer from:

  1. Extreme Occlusion: Apples are often hidden behind leaves, branches, or other fruit.
  2. Variable Lighting: Dappled sunlight and deep shadows create inconsistent color profiles.
  3. Scale Issues: A single image might contain over 100 tiny apples, making standard object detectors struggle.

Furthermore, most previous research in this field used small, private datasets. This lack of a "gold standard" benchmark made it impossible to objectively compare which algorithms actually perform best in the field.

Methods

The authors established a rigorous pipeline for data collection and benchmarking:

  • Data Collection: They used a standard smartphone (Samsung Galaxy S4) to record video while walking through the University of Minnesota’s Horticultural Research Center. This ensures the data reflects "real-world" low-cost sensor input.
  • Annotation: Instead of simple bounding boxes, they used polygonal masks for every fruit. This allows the dataset to support both object detection (finding the box) and instance segmentation (identifying the exact pixels of the fruit).
  • Baselines: They evaluated several state-of-the-art architectures to set a performance floor, including Faster R-CNN for detection, Mask R-CNN for instance segmentation, and UNet for semantic segmentation.
  • Counting: They introduced a patch-based counting method, comparing a traditional Gaussian Mixture Model (GMM) against a ResNet-50 based Convolutional Neural Network (CNN).

Datasets

The paper introduces the MinneApple dataset, which is significantly more dense than previous efforts:

  • Detection/Segmentation: 1,000 images containing 41,325 annotated instances.
  • Counting: A separate dataset of 13,000 image patches for training and nearly 3,000 for testing, specifically designed to handle clustered fruit.
  • Comparison: While the COCO dataset has an "apple" category, it averages only 7.7 instances per image. MinneApple averages 41.2 instances per image, with some images containing up to 120 apples.

Key Insights and Ideas

  • The "Small Object" Hurdle: The average apple size in the dataset is only about 40x40 pixels (0.17% of the total image area). The authors found that standard detectors struggle significantly with these small instances, identifying this as the primary area for future architectural improvement.
  • Tiling Isn't a Silver Bullet: One common trick for small objects is "tiling" (chopping the image into smaller chunks). Surprisingly, the authors found that a Tiled Faster R-CNN performed worse than the standard version. This was likely due to artifacts created at the tile boundaries and issues with merging overlapping detections (Non-Maximum Suppression).
  • Pretraining Matters: Even though orchards look nothing like the typical images in ImageNet, models pretrained on ImageNet significantly outperformed those trained from scratch. This suggests that the "low-level" features (edges, textures) learned from general data are highly transferable to agricultural contexts.

Key Results

  • Detection: Faster R-CNN achieved the best performance with an Average Precision (AP) of 0.438.
  • Segmentation: A pretrained UNet model led the pack with an Intersection over Union (IoU) of 0.685.
  • Counting: The CNN-based approach was highly robust, achieving over 90% accuracy in most test cases, far outperforming the unsupervised GMM method.
  • Yield Estimation: When combining detection and tracking, the system achieved between 95.5% and 97.8% accuracy compared to the actual ground-truth harvest count.

Limitations and Future Work

While MinneApple is a major step forward, the authors note a few remaining challenges:

  • Tracking: The current benchmark focuses on static images. To truly estimate yield, the community needs better "multi-object tracking" (MOT) to ensure the same apple isn't counted twice as the camera moves down a row.
  • Class Imbalance: There is a massive disparity between "background" pixels (leaves, sky, dirt) and "apple" pixels. Future work could involve specialized loss functions to force the network to pay more attention to the fruit.
  • Low-Resolution Fruit: Detecting fruit that is far away or blurred by motion remains a difficult task that current models haven't fully solved.

Summary updated

Abstract

In this work, we present a new dataset to advance the state-of-the-art in fruit detection, segmentation, and counting in orchard environments. While there has been significant recent interest in solving these problems, the lack of a unified dataset has made it difficult to compare results. We hope to enable direct comparisons by providing a large variety of high-resolution images acquired in orchards, together with human annotations of the fruit on trees. The fruits are labeled using polygonal masks for each object instance to aid in precise object detection, localization, and segmentation. Additionally, we provide data for patch-based counting of clustered fruits. Our dataset contains over 41'0000 annotated object instances in 1000 images. We present a detailed overview of the dataset together with baseline performance analysis for bounding box detection, segmentation, and fruit counting as well as representative results for yield estimation. We make this dataset publicly available and host a CodaLab challenge to encourage a comparison of results on a common dataset. To download the data and learn more about the MinneApple dataset, please see the project website: http://rsn.cs.umn.edu/index.php/MinneApple. Up to date information is available online.
BibTeX
@article{hani2020minneapple,
  author = {Nicolai Häni and Pravakar Roy and Volkan Isler},
  title = {{MinneApple: A Benchmark Dataset for Apple Detection and Segmentation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  url = {https://openreview.net/forum?id=2HA1ShhKFc},
}

Loading related content...

Figure