BFP Net: Balanced Feature Pyramid Network for Small Apple Detection in Complex Orchard Environment

Meili Sun, Liancheng Xu, Xiude Chen, Ze Ji, Yuanjie Zheng, Weikuan JiaPlant 2022 · DOI

TL;DR

This paper introduces BFP Net, a novel Balanced Feature Pyramid Network designed to tackle the persistent challenge of detecting small apples in complex orchard environments. By introducing a weight-like feature fusion and a decoupled-aggregated module, BFP Net effectively balances multi-scale information and enhances spatial localization, leading to improved detection accuracy for small fruits.

What Problem Does This Paper Address?

Detecting small objects, particularly fruits in their early growth stages, is a significant hurdle in agricultural computer vision. General object detection models often struggle with small targets due to their limited pixel representation and the difficulty in distinguishing them from cluttered backgrounds, especially when fruit color closely matches foliage. This leads to a substantial drop in precision for small objects compared to larger ones, hindering applications like automated harvesting and yield prediction. The paper specifically targets the challenge of detecting small green apples, which are particularly hard to identify due to their size and color similarity to their surroundings.

Methods

The core of the proposed solution is the Balanced Feature Pyramid Network (BFP Net), built upon the Faster R-CNN architecture. BFP Net introduces several key innovations:

  • Weight-like Feature Fusion: Standard Feature Pyramid Networks (FPNs) fuse features from different layers using simple element-wise addition. BFP Net replaces this with a "weight-like" fusion mechanism. This approach assigns different weights to features from various layers, allowing for a more balanced contribution of information from both shallow (detail-rich) and deep (semantic-rich) layers, thereby better preserving information relevant to small apples.
  • Extended Feature Layer: A new extended feature layer is introduced before the standard FPN's pooling operation. This layer, derived from an earlier stage of the ResNet50 backbone, retains higher resolution and thus more spatial information, which is crucial for localizing small objects.
  • Decoupled-Aggregated Module: This module is applied to the new extended feature layer. It decouples the feature into spatial-aware and content-aware branches. The spatial branch focuses on preserving location information, while the content branch captures global context. These are then aggregated to create a richer feature representation that aids in distinguishing small apples from the background.
  • Kullback-Leibler (KL) Distillation Loss: To further improve feature consistency between the extended layer and the standard FPN, a KL distillation loss is employed. This transfers knowledge from a "teacher" (standard FPN features) to a "student" (extended layer features), ensuring a more cohesive representation.

Datasets

The proposed BFP Net was evaluated on three datasets:

  • GreenApple: A newly constructed dataset specifically for small green fruit detection and segmentation, containing 1361 images and 7137 apple instances. This dataset is notable for its focus on challenging scenarios with varying illumination and fruit patterns.
  • MinneApple: A public dataset for apple detection and segmentation, used to validate the method's effectiveness.
  • Pascal VOC: A widely used generic object classification and detection dataset, employed to assess the model's generalization capabilities.

Key Insights and Ideas

  • The "weight-like" fusion is crucial: Simple, equal-weight fusion in FPNs is suboptimal for small objects because it doesn't account for the varying importance of features at different scales. BFP Net's weighted approach directly addresses this imbalance.
  • Early features are gold for small objects: The paper highlights that shallow features contain more information relevant to small apples. By introducing an extended layer before pooling, BFP Net preserves this valuable high-resolution information.
  • Spatial awareness is key for localization: The decoupled-aggregated module's spatial-aware branch is particularly effective because it retains precise location information, which is vital for distinguishing tiny objects from their surroundings.
  • Generalization is achievable: The model's strong performance on Pascal VOC demonstrates that its improvements for small apple detection are not just dataset-specific but can generalize to broader object detection tasks.

Key Results

BFP Net demonstrated significant improvements across the evaluated datasets:

  • GreenApple: Achieved an AP of 47.0% for small apples, an improvement of 1.3% over the Faster R-CNN baseline.
  • MinneApple: Reached an AP of 42.2% for small apples, outperforming the baseline by 1.0%.
  • Pascal VOC: Achieved an AP of 35.6% for small objects, a substantial increase of 8.5% over the Faster R-CNN baseline.
  • Qualitative results showed BFP Net's ability to accurately detect small apples even in challenging conditions with similar fruit and background colors, and occlusions.
  • The model achieved these improvements without a significant increase in parameters, with only a 1.43M parameter increase compared to Faster R-CNN.

Limitations and Future Work

  • Dataset Specificity: While BFP Net shows promise for small apple detection, its direct applicability to other small fruit types or significantly different orchard environments might require further validation.
  • General Object Detection Suitability: The paper notes that while BFP Net improves small apple detection, existing detectors are not always perfectly suited for detecting fruits across all scales throughout their growth.
  • Future Directions: The authors suggest that optimizing fruit detectors specifically for small fruits and exploring how to balance phenotyping traits from multiple perspectives remain challenging but important areas for future research. They also propose that BFP Net can serve as a reference for other fruit/vegetable detection tasks.

Summary updated

Abstract

Despite of significant achievements made in the detection of target fruits, small fruit detection remains a great challenge, especially for immature small green fruits with a few pixels. The closeness of color between the fruit skin and the background greatly increases the difficulty of locating small target fruits in the natural orchard environment. In this paper, we propose a balanced feature pyramid network (BFP Net) for small apple detection. This network can balance information mapped to small apples from two perspectives: multiple-scale fruits on the different layers of FPN and a characteristic of a new extended feature from the output of ResNet50 conv1. Specifically, we design a weight-like feature fusion architecture on the lateral connection and top-down structure to alleviate the small-scale information imbalance on the different layers of FPN. Moreover, a new extended layer from ResNet50 conv1 is embedded into the lowest layer of standard FPN, and a decoupled-aggregated module is devised on this new extended layer of FPN to complement spatial location information and relieve the problem of locating small apple. In addition, a feature Kullback-Leibler distillation loss is introduced to transfer favorable knowledge from the teacher model to the student model. Experimental results show that AP<sub>S</sub> of our method reaches 47.0%, 42.2%, and 35.6% on the benchmark of the GreenApple, MinneApple, and Pascal VOC, respectively. Overall, our method is not only slightly better than some state-of-the-art methods but also has a good generalization performance.
BibTeX
@inproceedings{sun2022bfp,
  author = {Meili Sun and Liancheng Xu and Xiude Chen and Ze Ji and Yuanjie Zheng and Weikuan Jia},
  title = {{BFP Net: Balanced Feature Pyramid Network for Small Apple Detection in Complex Orchard Environment}},
  booktitle = {Plant},
  year = {2022},
  doi = {10.34133/2022/9892464},
}
Figure