U-Net-Based Foreign Object Detection Method Using Effective Image Acquisition System: A Case of Almond and Green Onion Flake Food Process

Guk-Jin Son, Donghoon Kwak, Mi‐Kyung Park, Young‐Duk Kim, Heechul JungSustainability 2021 · DOI

TL;DR

This paper introduces a novel U-Net-based deep learning approach for foreign object detection in food processing. By leveraging a custom image acquisition system and a synthetic data generation method, they significantly reduce the need for manual annotation, achieving competitive performance with real datasets and demonstrating improved robustness to lighting variations.

What Problem Does This Paper Address?

Ensuring food safety and quality often involves detecting foreign objects in food products. Traditional methods, like color sorters, struggle with objects similar in color to the food and are highly sensitive to lighting changes. Supervised deep learning methods, while promising, typically require massive, meticulously annotated datasets, which are costly and time-consuming to create, especially for food inspection where foreign objects can be diverse and unpredictable. This paper aims to bridge this gap by developing a more efficient and robust deep learning-based foreign object detection (FODM) system that minimizes the reliance on extensive manual annotation and improves performance under varying illumination conditions.

Methods

The core of the proposed method is a U-Net architecture, a popular choice for semantic segmentation tasks, adapted for foreign object detection. The key innovations lie in:

  • Effective Image Acquisition System: A specialized system combining reflectance and transmittance illumination modes is used to capture images of raw materials of food (RMF) and foreign objects. This setup is designed to minimize shadows and clearly distinguish foreground (RMF) from background, facilitating easier segmentation.
  • Synthetic Data Generation: Instead of relying solely on real, manually annotated data, the authors generate synthetic training datasets. This involves pasting RMF images (acquired from their system) onto backgrounds from the public Food101 dataset. The binarized RMF masks generated by the Otsu algorithm serve as automatic annotations.
  • Data Augmentation: Color jittering (randomly adjusting saturation, contrast, and brightness) is applied to the synthetic dataset. This is crucial for improving the model's invariance to illumination changes.
  • U-Net for RMF Prediction: A U-Net model is trained to predict the mask of the RMF. This is a crucial step as the system focuses on identifying RMF and background, with foreign objects being what remains.
  • Background Estimation and Subtraction: The background is estimated, and then subtracted along with the predicted RMF mask from the original image to isolate potential foreign objects.

Datasets

  • Real Datasets: The paper uses real images of almonds and green onion flakes (GOF) acquired using their custom imaging system. These datasets are used for both training and testing, with manual annotations for ground truth.
  • Synthetic Datasets: Generated by combining RMF images (from their system) with backgrounds from the Food101 dataset. The Food101 dataset is a large-scale dataset containing 101 food categories and 101,000 images, used here to provide diverse backgrounds. The synthetic images are automatically annotated using Otsu's method.
  • Test Datasets: Real images of almonds and GOF are used for testing, acquired under varying illumination intensities (360 lux, 550 lux, and 175 lux) to evaluate robustness.

Key Insights and Ideas

  • Synthetic Data as a Viable Alternative: The study demonstrates that synthetic datasets, when generated with an effective acquisition system and appropriate augmentation, can achieve performance comparable to, and sometimes superior to, real, manually annotated datasets. This significantly reduces the cost and effort of data preparation.
  • Illumination Invariance is Key: The color jittering applied to synthetic data proved highly effective in making the trained model robust to changes in illumination, a common challenge for traditional methods.
  • Focusing on RMF Simplifies the Problem: By training the model to identify the RMF and background, the problem of detecting all possible foreign objects is reframed. Foreign objects are implicitly identified as what's left after RMF and background are accounted for.
  • Complementary Nature of Real and Synthetic Data: Combining real and synthetic data for training yielded the best overall performance, suggesting that each dataset type offers unique advantages that complement the other.

Key Results

  • Almond Detection: The U-Net model trained on synthetic almond data achieved an F1-score of 0.82 at 360 lux, comparable to the 0.82 achieved with real data.
  • Illumination Robustness: Models trained on synthetic data showed significantly less performance degradation under varying illumination (175 lux and 550 lux) compared to models trained solely on real data. For instance, for almonds at 175 lux, synthetic training achieved an F1-score of 0.80, while real training achieved 0.78.
  • Combined Training: Training with both real and synthetic data further improved performance, achieving an F1-score of 0.81 for almonds under varying illumination.
  • Comparison to HBFOD: The proposed Deep Learning-based Foreign Object Detection (DLFOD) method consistently outperformed the traditional Histogram Backprojection (HBFOD) method, especially for objects with similar colors to the RMF and under varying lighting conditions. For example, for GOF at 175 lux, DLFOD achieved an F1-score of 0.70, while HBFOD achieved only 0.53.

Limitations and Future Work

  • Object Viewpoint and Scale: The current method assumes objects have similar viewpoints and scales, and that the background is relatively monotonous. This limits its applicability to general object detection but is suitable for controlled food inspection environments.
  • Overlapping and Thin Objects: Detecting highly overlapping or very thin foreign objects (like some GOF) remains a challenge and requires further investigation.
  • Future Work: The authors suggest exploring multi-waveband imaging hardware to acquire datasets with richer features for RMF, which could further enhance foreign object detection performance. They also propose investigating DNNs for detecting thin and overlapping RMF.

Summary updated

Abstract

Supervised deep learning-based foreign object detection algorithms are tedious, costly, and time-consuming because they usually require a large number of training datasets and annotations. These disadvantages make them frequently unsuitable for food quality evaluation and food manufacturing processes. However, the deep learning-based foreign object detection algorithm is an effective method to overcome the disadvantages of conventional foreign object detection methods mainly used in food inspection. For example, color sorter machines cannot detect foreign objects with a color similar to food, and the performance is easily degraded by changes in illuminance. Therefore, to detect foreign objects, we use a deep learning-based foreign object detection algorithm (model). In this paper, we present a synthetic method to efficiently acquire a training dataset of deep learning that can be used for food quality evaluation and food manufacturing processes. Moreover, we perform data augmentation using color jitter on a synthetic dataset and show that this approach significantly improves the illumination invariance features of the model trained on synthetic datasets. The F1-score of the model that trained the synthetic dataset of almonds at 360 lux illumination intensity achieved a performance of 0.82, similar to the F1-score of the model that trained the real dataset. Moreover, the F1-score of the model trained with the real dataset combined with the synthetic dataset achieved better performance than the model trained with the real dataset in the change of illumination. In addition, compared with the traditional method of using color sorter machines to detect foreign objects, the model trained on the synthetic dataset has obvious advantages in accuracy and efficiency. These results indicate that the synthetic dataset not only competes with the real dataset, but they also complement each other.
BibTeX
@inproceedings{son2021u,
  author = {Guk-Jin Son and Donghoon Kwak and Mi‐Kyung Park and Young‐Duk Kim and Heechul Jung},
  title = {{U-Net-Based Foreign Object Detection Method Using Effective Image Acquisition System: A Case of Almond and Green Onion Flake Food Process}},
  booktitle = {Sustainability},
  year = {2021},
  doi = {10.3390/su132413834},
}
Figure