Semmeldetector: Application of Machine Learning in Commercial Bakeries

Thomas H. Schmitt, Maximilian Bundscherer, Tobias Bocklet

Abstract

The Semmeldetector project introduces a machine learning pipeline tailored for commercial bakeries to automate the tracking of unsold inventory. Utilizing the YOLOv8 architecture and a novel "Copy-Paste" data augmentation strategy, the system achieves an $AP_{50}$ of 89.1% across 18 distinct bread and pastry categories. This work demonstrates the feasibility of deploying computer vision in low-data, high-variability industrial settings to reduce food waste and optimize production.

1. Introduction

In the German bakery industry, inventory management is complicated by a vast diversity of products. Currently, tracking unsold goods at the end of the day is a manual, labor-intensive process. Automating this task would allow bakeries to optimize production cycles and significantly reduce food waste. However, two primary challenges hinder automation: a lack of public datasets for fine-grained baked good classification and high environmental variability in bakery branches, such as inconsistent lighting and crowded displays.

2. Methodology

The researchers developed a detection pipeline centered on the YOLOv8 architecture. To overcome the data bottleneck, the team employed several innovative strategies:

  • Automated Annotation: The Segment Anything Model (SAM) was used to automatically generate high-quality segmentation masks, bypassing the need for manual labeling.
  • Copy-Paste Augmentation: To simulate realistic, crowded bakery trays from single-item images, a "Copy-Paste" pipeline was implemented. This involved extracting precise pixels of baked goods and pasting them onto various backgrounds, including random images from the DIV2K dataset, to model occlusion and density.
  • Grayscale Processing: Converting images to grayscale was found to improve model robustness against the varied lighting conditions found in different retail environments.
Examples of the Copy-Paste augmentation pipeline used to create synthetic training data.

3. Dataset

The study introduced a specialized dataset comprising 1,151 baseline images across 18 types of baked goods. This baseline was expanded into a final training set of 6,897 images through synthetic generation. The test set (209 images) was specifically curated with "unseen" samples collected on different days and with different hardware (e.g., iPad vs. HD webcams) to ensure the model's real-world generalization capabilities.

4. Results and Analysis

The YOLOv8-X model emerged as the top performer, achieving an 89.1% $AP_{50}$ on the test set. Key insights from the evaluation include:

  • Resolution Gains: Increasing input resolution to 1280px significantly improved performance, as fine details like seeds are critical for distinguishing similar bun types.
  • Robustness: The model maintained high accuracy even when detecting "broken" or crumbled goods, which is essential for waste tracking.
  • Fine-Grained Challenges: Despite high overall accuracy, the model faces challenges with extremely similar classes. For instance, the Sesamstange (sesame breadstick) and Kaisersemmel (emperor bun) are frequently confused, particularly when photographed from non-standard angles.
Normalized confusion matrix showing the model's performance across 18 categories and the background class.

5. Conclusion and Future Work

The Semmeldetector proves that modern object detection can be effectively adapted for traditional industrial sectors. Future work will focus on scaling the assortment beyond 18 categories and exploring generative AI (GANs or Diffusion models) to further automate the synthesis of realistic training data. Additionally, the team plans to evaluate newer architectures like YOLOv7 and Swin-Transformers to address remaining fine-grained classification errors.

Summary updated

BibTeX
@inproceedings{schmitt2023semmeldetector,
  author = {Thomas H. Schmitt and Maximilian Bundscherer and Tobias Bocklet},
  title = {{Semmeldetector: Application of Machine Learning in Commercial Bakeries}},
  booktitle = {ICMLA},
  year = {2023},
  doi = {10.1109/icmla58977.2023.00129},
}
Figure