Abstract
In this paper, we propose a new approach to estimating waiting time in queue based on object tracking and person re-identification by upper body. The task we are considering is practically important in video analysis, because this data can be used for predictive analytics and improvement of customer services. The main idea of the proposed method is to use upper body detections instead of full body detections. This decision is due to the following fact: in queues, the upper bodies are more visible. Using re-identification allows us to perform video analytics on sparse frames and thereby increase the computational efficiency of the estimation algorithm. Also in this work, we introduce a novel upper body regression by head, upper body random size augmentation to improve re-identification performance in real-world scenarios and a method for calculating metrics for queue waiting time estimation algorithms. Our experimental evaluation showed that the proposed algorithm has a high accuracy of queue waiting time estimation.
Introduction
Estimating customer wait times is a critical component of retail analytics. Traditional computer vision methods typically rely on full-body object tracking. However, these methods face significant challenges in crowded environments due to occlusions—where only heads and shoulders are visible—and high computational costs associated with processing every video frame.
This research introduces a more efficient and accurate framework by focusing on the upper body. By combining head detection with a specialized upper-body regression model and robust re-identification (Re-ID) techniques, the system achieves high accuracy even on sparse video frames, significantly reducing the computational power required for real-world deployment.
Methodology
The proposed pipeline consists of five distinct stages designed to handle the complexities of queue environments.
- Head Detection: Utilizing an SSD (Single Shot Multibox Detector) with a ResNet50 backbone, the system identifies heads, which are rarely occluded in queues.
- Upper Body Regression: Since heads lack sufficient detail for unique identification, the system estimates the upper body area. It begins with anatomical heuristics and refines the boundaries using a lightweight MobileNetV2 neural network.
- Re-ID Appearance Embeddings: A "visual fingerprint" is extracted using a Res2Net50 backbone and FIDI Loss, which effectively penalizes small visual differences to improve identification accuracy.
- Tracking: To optimize performance, the system links detections across sparse frames (e.g., every 5-10 seconds) using the Hungarian algorithm to match Re-ID fingerprints.
- Waiting Time Estimation: Final wait times are calculated based on the maximum "lifetime" of these tracks within specific video segments.
Upper Body Random Size Augmentation (UBRSA)
A key innovation in this work is Upper Body Random Size Augmentation (UBRSA). During the training phase, the system randomly shifts the bounding boxes of upper bodies. This technique trains the model to remain robust even when detection is slightly imprecise or when a person is partially obscured.
Experimental Setup and Datasets
The models were trained and evaluated using a combination of public and private datasets:
- CrowdHuman: Used for training the head detector and the upper body regression model.
- MSMT17: A large-scale person Re-ID dataset, modified to focus on upper bodies.
- Private Video Collection: Six long-duration videos (18 to 84 minutes) from real-world security cameras in retail and public spaces were used for final validation.
Results and Discussion
The upper-body approach was benchmarked against a standard full-body baseline using the Detectron2 framework. The results demonstrate a significant improvement in both accuracy and reliability:
- Accuracy: At a 5-second frame interval, the full-body baseline yielded a Mean Absolute Error (MAE) of 247.5 seconds. The proposed upper-body method with UBRSA reduced this error to just 23.1 seconds.
- Reliability: The Mean Relative Error (MRE) for the proposed model was 6.3%, compared to over 34% for the baseline.
- Efficiency: The system maintained high accuracy (MAE of 29.4s) even when processing only one frame every 10 seconds, proving its viability for low-resource hardware.
The "upper body" serves as a sweet spot for identification; while full bodies are often occluded and heads lack detail, the chest and shoulders provide sufficient visual information (color, texture) for reliable Re-ID.
Conclusion
This paper presents a robust and computationally efficient method for queue waiting time estimation. By leveraging upper-body Re-ID and sparse frame processing, the system overcomes common hurdles like occlusion and high processing costs. Future work will focus on improving dependency on head detection in poor lighting and adapting the system for non-stationary cameras.
Summary updated