TL;DR
This paper introduces StandardSim, a large-scale synthetic dataset for retail environments, designed to address the limitations of existing datasets for autonomous checkout systems. It also proposes a novel "change detection" task, crucial for understanding shopper interactions, and benchmarks existing models on these challenges.
What Problem Does This Paper Address?
Autonomous checkout systems rely heavily on computer vision to understand complex retail environments. However, these environments pose unique challenges: a vast number of densely packed, yet visually similar objects. Existing datasets often lack the fine-grained annotations (like pixel-level masks) or the specific domain focus needed for tasks like instance segmentation, especially when only RGB data is available. Furthermore, there's a lack of datasets and tasks specifically designed to capture the dynamic nature of retail, such as a shopper interacting with shelves (taking, putting, or shifting items). This gap hinders the development and benchmarking of robust autonomous checkout systems.
Methods
The authors introduce StandardSim, a synthetic dataset generated using Blender. Key aspects of their methodology include:
- Photorealistic Rendering: High-quality RGB images, depth maps, surface normals, and semantic/instance segmentation masks are generated.
- Realistic Retail Environments: The dataset is built upon 3D scans of real retail stores, ensuring accurate dimensions and structure.
- Diverse Object Placement: A large collection of 3D product models (SkuBank) is used to populate shelves in a dense, randomized manner, mimicking real-world retail.
- Multi-View Data: Each scene is rendered from multiple camera viewpoints, enabling multi-view representation learning and 3D reconstruction.
- Novel "Change Detection" Task: This task involves providing pairs of images (before and after shopper interaction) and requires pixel-level classification of "no change," "take," "put," or "shift" actions. This is achieved by simulating realistic object removal and subsequent shifting of nearby items.
- Benchmarking: Standard computer vision models for segmentation (Deeplabv3) and monocular depth estimation (DPT-Large) are benchmarked on StandardSim to establish its difficulty and utility.
Datasets
- StandardSim (New Dataset):
- Contains 2,134 unique retail scenes based on three real store layouts.
- Features 25,436 images in total, structured as 6,359 "change detection" samples (pairs of before/after images).
- Provides annotations for: semantic segmentation, instance segmentation, depth estimation, object detection, and the novel change detection task.
- Offers multiple camera views per scene (average of 3).
- A 70/15/15 train/validation/test split is used.
- Existing Datasets Mentioned for Comparison:
- SKU-110K: Retail dataset with bounding box annotations, but lacks semantic masks.
- Hypersim, SceneNet: Photorealistic indoor datasets, but not specifically focused on retail environments.
- ChangeSim: An indoor change detection dataset, but focuses on industrial environments and uses fixed cameras.
- NYUv2: Indoor dataset used for depth estimation benchmarking.
- Sintel, TUM, Kitti: Datasets used for monocular depth estimation benchmarking.
Key Insights and Ideas
- Retail is a Unique Domain: The dense packing and similarity of objects in retail environments present a distinct challenge compared to general indoor or outdoor scenes, requiring specialized datasets.
- Change Detection is Crucial: Understanding shopper actions (takes, puts, shifts) is fundamental for autonomous checkout, and this dynamic aspect is not well-addressed by existing datasets or tasks.
- Synthetic Data Can Bridge the Gap: StandardSim demonstrates that high-fidelity synthetic data can be generated to provide rich annotations and address specific domain challenges where real-world data is scarce or difficult to annotate.
- StandardSim is a Challenging Benchmark: The paper shows that state-of-the-art models struggle on StandardSim, particularly for change detection and depth estimation, highlighting the dataset's difficulty and its potential to drive research.
Key Results
- Change Detection Performance: A Deeplabv3 model trained on StandardSim achieved 36.04% IOU on the test set. This is considered a difficult task, with the model performing best on "put" actions and struggling with "shift" actions. Qualitative results show the model can detect changes even on small objects.
- Depth Estimation Performance: The DPT-Large model, when benchmarked on StandardSim, showed significantly higher errors (Absolute Relative Error of 83.59) compared to other datasets like NYUv2 (8.43). This indicates StandardSim is a more challenging benchmark for monocular depth estimation.
- Synthetic-to-Real Transfer: Qualitative results suggest that models trained on StandardSim can generalize to real retail change detection scenarios, indicating the dataset's potential for domain adaptation.
Limitations and Future Work
- Limitations:
- The authors acknowledge that the current dataset is synthetic and domain transfer to real-world scenarios, while showing promise, still has challenges.
- The dataset's depth estimation performance indicates difficulties with reflective surfaces and fine structures, and challenges in distinguishing small depth changes between small objects.
- Some scenes have occlusions due to lighting fixtures, leading to zero ground truth depth, which requires masking for metric calculations.
- Future Work:
- The paper suggests that the provided multi-view data can be used for multiview 3D reconstruction, which has applications in 3D object detection, instance segmentation, and understanding store layout changes.
- The dataset can serve as a valuable augmentation for existing depth estimation datasets to improve performance on small and distant objects.
- The authors plan to make the dataset publicly available.
Summary updated