Abstract
In retail analytics, accurately tracking individuals and distinguishing between customers and staff is critical for optimizing store layouts and preventing fraud. However, traditional Multiple Object Tracking (MOT) systems often fail in crowded supermarket environments due to frequent occlusions and the visual similarity between shoppers and employees. This paper proposes a novel two-stage framework that utilizes head detection to minimize occlusion errors and converts movement trajectories into 2D heatmaps for role classification. Using a ResNet-50 architecture, the system achieves a 93% accuracy rate in distinguishing staff from customers in real-world settings.
Introduction
Understanding the flow of people within a retail space provides invaluable insights for store management, from identifying high-traffic "hot zones" to detecting suspicious behavior. Despite the potential, computer vision in retail faces significant hurdles. Supermarkets are characterized by high-density crowds, narrow aisles, and tall shelving units that frequently block a camera's line of sight. Furthermore, identifying the role of an individual—whether they are a shopper or a staff member—is difficult when uniforms are absent or camera resolutions are low.
Researchers at the Sunmi AI Lab address these challenges by shifting the focus from full-body tracking to head-centric detection and leveraging spatial-temporal movement patterns as a "role biometric."
Methodology
The proposed system operates via a two-stage pipeline designed for robustness in high-occlusion environments.
1. Head-Centric Tracking
Traditional MOT systems typically rely on full-body bounding boxes. In a retail aisle, however, the lower body is often obscured by shopping carts or shelving. By mounting cameras on the ceiling and focusing exclusively on head detection, the system significantly reduces the impact of these occlusions. The researchers evaluated two primary detectors: YOLOv3 and SSD ResNet-50. Once heads are detected, the DeepSORT algorithm is employed to link these detections across frames, maintaining consistent identity (ID) tags even as individuals move through the store.
2. Trajectory-to-Heatmap Classification
A key innovation of this research is the method used to classify the "role" of the tracked individual. Rather than using computationally expensive temporal models like Long Short-Term Memory (LSTM) networks to analyze movement frame-by-frame, the system collapses a person's entire trajectory into a single 2D image: a track heatmap.
In these heatmaps, pixel intensity represents "exposure time"—the duration an individual remained at a specific coordinate. These images are then processed by a standard ResNet-50 image classifier. This approach capitalizes on the fact that staff and customers exhibit distinct spatial behaviors; for instance, staff members often show high exposure times behind checkout counters, while customers show more distributed paths through the aisles.
Experimental Setup and Datasets
The study utilized two distinct datasets to validate the system:
- In-Office Dataset: A controlled environment where participants performed specific supermarket-related behaviors—such as squatting to reach low shelves, wearing hats, or moving in tight groups—to test the robustness of the head-tracking algorithm.
- Supermarket Dataset: A large-scale, real-world dataset consisting of 24 hours of video from a functioning supermarket in China. This dataset contained over 11,000 individual tracks and served as the primary training and testing ground for the customer-vs-staff classification model.
Key Results and Insights
Detection and Tracking Performance
The experimental results indicated that YOLOv3 was more effective than SSD ResNet-50 for head detection in the specific context of retail surveillance. Notably, the system demonstrated high resilience to data loss; the tracking remained accurate even when up to 30% of video frames were skipped, suggesting the model could be deployed on edge devices with limited processing power.
Classification Accuracy
The heatmap-based classification proved highly effective. The ResNet-50 model achieved a 98% accuracy during the training phase and maintained a 93% accuracy on the test set. This confirms that movement patterns alone, when visualized as heatmaps, are a powerful indicator of a person's role within a commercial environment.
The "Role Biometric"
The research highlights that movement is a reliable proxy for identity in specialized environments. Because a staff member's job functions dictate specific movement signatures (e.g., stationary periods at registers or repetitive restocking paths), the system can identify them without needing facial recognition or uniform detection, preserving a degree of privacy while providing actionable data.
Limitations and Future Work
While the system is highly accurate for long-duration tracks, it struggles with "short tracks"—individuals who appear in the camera's view for only a brief period. The researchers found that approximately 31% of initial tracks were erroneous, often caused by inanimate objects being misidentified as heads. To maintain high accuracy, the current model requires a minimum track length of 2,000 frames.
Future iterations of this work aim to refine the handling of short trajectories and expand the classification capabilities. Beyond simple role identification, the researchers hope to detect specific behaviors, such as a customer needing assistance or the occurrence of shoplifting, by further analyzing the nuances of the generated heatmaps.
Summary updated