Robust Multi-Object Detection Based on Data Augmentation with Realistic Image Synthesis for Point-of-Sale Automation

Saiprasad Koturwar, Soma Shiraishi, Kota IwamotoAAAI 2019 · DOI

Introduction

The retail industry is increasingly adopting automated, "scan-less" checkout systems to address labor shortages and minimize customer wait times. A significant hurdle for these systems is the point-of-sale (POS) environment, where customers naturally place multiple items on a counter in close proximity. These items often touch or partially overlap, creating "congested" scenes that standard deep learning-based object detectors struggle to process accurately.

Training robust models for these scenarios typically requires massive, manually annotated datasets of various product combinations, which is both labor-intensive and cost-prohibitive. While synthetic data generation offers a potential solution, traditional "cut-and-paste" methods often lack the realism or control necessary to prepare a model for the complexities of a real-world checkout counter.

Methodology

Researchers at NEC Corporation have proposed a novel two-stage pipeline (Detection followed by Classification) that leverages a sophisticated data augmentation technique to synthesize realistic training images.

Hardware-Software Co-design for Masking

To create high-quality synthetic data, the system requires precise pixel-level masks of individual products. The researchers developed a clever, low-cost solution using an LCD screen as a dynamic background. By placing a product on the screen and cycling through multiple background colors while a camera captures images, the system can calculate the pixel-wise standard deviation. Static product pixels show low deviation, while the changing background pixels show high deviation, allowing for the automated extraction of perfect product masks without the limitations of traditional green screens.

The automated POS checkout setup featuring an overhead camera and an LCD screen for item placement and mask extraction.

Realistic Synthesis with the Overlap Index (OI)

Rather than scattering products randomly, the proposed method uses a controllable parameter called the Overlap Index (OI), based on Intersection over Union (IoU). This allows the system to simulate varying degrees of item congestion. The study found that an $OI = 0.01$—representing slight touching or minimal overlap—serves as the "sweet spot" for mimicking natural human behavior at a checkout, providing enough complexity to challenge the model without making the training data unintelligible.

Experimental Evaluation

The researchers trained a Faster R-CNN (with a ResNet-101 backbone) using several datasets:

  • Non-augmented: 1,638 real images of single products.
  • Simple cut-and-paste: 20,000 basic synthetic images.
  • Proposed Synthetic: 20,000 images generated with optimized Overlap Indices.

Testing was conducted on real-world (non-synthesized) images categorized by difficulty: Easy (separated items), Regular (touching items), and Difficult (highly congested items).

Results and Discussion

The proposed synthesis method demonstrated a dramatic improvement in detection performance, particularly in congested scenarios. On the "Regular" test set, precision increased from 0.67 to 0.98 (a 46.2% improvement) compared to the non-augmented baseline.

Example of the detector successfully identifying and bounding multiple congested items on the POS counter.

In the "Difficult" test set, where the baseline model essentially failed with a precision of 0.03, the proposed method maintained a robust 0.84 precision and 0.61 recall. Furthermore, the Average Overlap with Ground Truth (AOGB) remained consistently high (around 0.85), indicating superior localization accuracy across all levels of scene complexity.

Conclusion

By combining automated mask extraction via LCD screens with a controllable overlap index for image synthesis, this research provides a scalable path toward robust POS automation. The decoupling of detection from classification ensures that the system remains reliable even when items are crowded. Future work will focus on simulating complex lighting and shadows to further enhance synthesis realism and exploring detection limits under extreme occlusion.

Summary updated

BibTeX
@inproceedings{koturwar2019robust,
  author = {Saiprasad Koturwar and Soma Shiraishi and Kota Iwamoto},
  title = {{Robust Multi-Object Detection Based on Data Augmentation with Realistic Image Synthesis for Point-of-Sale Automation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019492},
}
Figure