Application of Deep Learning Architectures for Accurate and Rapid Detection of Internal Mechanical Damage of Blueberry Using Hyperspectral Transmittance Data

Zhaodi Wang, Menghan Hu, Guangtao Zhai

TL;DR

This paper introduces deep learning, specifically ResNet and ResNeXt, for detecting internal mechanical damage in blueberries using hyperspectral transmittance data. The deep learning models significantly outperform traditional machine learning methods, achieving high accuracy and speed, paving the way for efficient, automated fruit quality inspection.

What Problem Does This Paper Address?

Blueberries, despite their economic value and nutritional benefits, are highly susceptible to mechanical damage during post-harvest handling. This damage, often invisible externally, accelerates softening and increases the risk of pathogen infection, leading to significant economic losses. Traditional methods for detecting this internal damage are either manual, time-consuming, and error-prone, or rely on less sensitive techniques. While hyperspectral imaging offers a promising non-destructive approach, effectively analyzing the complex hyperspectral data for internal damage detection, especially in agricultural applications, remains a challenge.

Methods

The researchers employed two state-of-the-art deep convolutional neural networks (CNNs): Residual Network (ResNet) and its enhanced version, ResNeXt. They adapted these architectures for hyperspectral transmittance data by adjusting the number of filters in their convolutional layers to accommodate the data's unique structure.

Key preprocessing steps for the hyperspectral data included:

  • Subsampling: Reducing the number of spectral channels from the original 1002 to 151 to manage computational complexity and avoid overfitting.
  • Resizing: Downscaling the image cubes to 32x32 pixels.
  • Data Augmentation: Expanding the training dataset by applying transformations like flipping and rotation to improve model robustness.
  • Normalization: Zero-centering and scaling the data per wavelength channel.

For comparison, five traditional machine learning algorithms were implemented: Sequential Minimal Optimization (SMO), Linear Regression (LR), Random Forest (RF), Bagging, and Multilayer Perceptron (MLP). Feature extraction for these models involved creating a mask from specific spectral bands to isolate the fruit and then calculating the average transmittance spectrum.

Model performance was evaluated using k-fold cross-validation (k=10) and metrics including accuracy, precision, recall, and F1-score. Receiver Operating Characteristic (ROC) curves and Precision-Recall curves were also generated for visualization.

Datasets

The study utilized a custom dataset of 737 blueberries collected from Chile. After cutting the berries equatorially to reveal internal damage (ground truth), the dataset comprised 557 samples: 304 sound and 253 damaged. The damaged category included berries with over 25% of their cut surface showing damage. This dataset was split into training and testing sets using a 10-fold cross-validation approach.

Key Insights and Ideas

  • Deep Learning's Potential in Agriculture: The paper strongly advocates for the adoption of deep learning in agriculture, demonstrating its superiority over traditional methods for a critical quality control task.
  • Hyperspectral Transmittance is Key: The study highlights hyperspectral transmittance imaging as a particularly effective mode for detecting internal damage, showing greater sensitivity than reflectance or interactance modes.
  • Architectural Adaptation is Crucial: Simply applying off-the-shelf deep learning models isn't enough. The researchers emphasize the need to adjust CNN architectures (like filter counts) to effectively handle the unique, high-dimensional nature of hyperspectral data.
  • Beyond Accuracy: The inclusion of precision, recall, and F1-score is vital for practical applications. The paper correctly identifies that missing a damaged berry (low recall) is more costly than misclassifying a good berry as damaged (low precision), making recall a particularly important metric.

Key Results

  • Superior Performance: Fine-tuned ResNet and ResNeXt achieved significantly better results than traditional methods.
  • ResNet: Average accuracy of 0.8844, F1-score of 0.8952, and recall of 0.9325.
  • ResNeXt: Average accuracy of 0.8784, F1-score of 0.8905, and recall of 0.8944.
  • Traditional Models: The best performing traditional model, SMO, achieved an average accuracy of 0.8082 and an F1-score of 0.8286, falling short of the deep learning models.
  • Speed Advantage: Classification for each testing sample was remarkably fast: 5.2 ms for ResNet and 6.5 ms for ResNeXt, making real-time, online sorting feasible.
  • Visualizations: ROC curves showed ResNet and ResNeXt achieving the highest AUC values (0.9248 and 0.9070, respectively), confirming their superior discriminative power. Precision-Recall curves further illustrated the deep learning models' ability to balance precision and recall effectively.

Limitations and Future Work

  • Local Correlation: The authors acknowledge that the local correlation between spectral channels in hyperspectral images is an area that traditional convolutional layers don't fully exploit. They suggest future work could explore specialized convolutional layers designed to handle this spectral correlation more effectively.
  • Feature Pre-extraction: To address variations across different production batches or locations, the paper proposes exploring feature pre-extraction methods to standardize samples before training.
  • Generalizability: While the models performed well on their dataset, further validation on larger, more diverse datasets from various sources would be beneficial to confirm generalizability.
  • Other Fruit Types: The success with blueberries suggests potential for applying similar deep learning approaches to detect internal damage in other soft fruits.

Summary updated

Abstract

Deep learning has become a widely used powerful tool in many research fields, although not much so yet in agriculture technologies. In this work, two deep convolutional neural networks (CNN), viz. Residual Network (ResNet) and its improved version named ResNeXt, are used to detect internal mechanical damage of blueberries using hyperspectral transmittance data. The original structure and size of hypercubes are adapted for the deep CNN training. To ensure that the models are applicable to hypercube, we adjust the number of filters in the convolutional layers. Moreover, a total of 5 traditional machine learning algorithms, viz. Sequential Minimal Optimization (SMO), Linear Regression (LR), Random Forest (RF), Bagging and Multilayer Perceptron (MLP), are performed as the comparison experiments. In terms of model assessment, k-fold cross validation is used to indicate that the model performance does not vary with the different combination of dataset. In real-world application, selling damaged berries will lead to greater interest loss than discarding the sound ones. Thus, precision, recall, and F1-score are also used as the evaluation indicators alongside accuracy to quantify the false positive rate. The first three indicators are seldom used by investigators in the agricultural engineering domain. Furthermore, ROC curves and Precision-Recall curves are plotted to visualize the performance of classifiers. The fine-tuned ResNet/ResNeXt achieve average accuracy and F1-score of 0.8844/0.8784 and 0.8952/0.8905, respectively. Classifiers SMO/ LR/RF/Bagging/MLP obtain average accuracy and F1-score of 0.8082/0.7606/0.7314/0.7113/0.7827 and 0.8268/0.7796/0.7529/0.7339/0.7971, respectively. Two deep learning models achieve better classification performance than the traditional machine learning methods. Classification for each testing sample only takes 5.2 ms and 6.5 ms respectively for ResNet and ResNeXt, indicating that the deep learning framework has great potential for online fruit sorting. The results of this study demonstrate the potential of deep CNN application on analyzing the internal mechanical damage of fruit.
BibTeX
@inproceedings{wang2018application,
  author = {Zhaodi Wang and Menghan Hu and Guangtao Zhai},
  title = {{Application of Deep Learning Architectures for Accurate and Rapid Detection of Internal Mechanical Damage of Blueberry Using Hyperspectral Transmittance Data}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=zHHHI_DhVxx},
}
Figure