Application of Openpose algorithm to detect consumer behavior in store

Jingjing Li, Jie Zeng, Keyu Hou, Jin Zhou, Rui Wang2020 · DOI

Abstract

Due to the importance of offline consumer behavior, more and more people had begun to study consumer behavior in store. In offline consumer behavior research, the application of video analysis technology was the most direct and convenient. Recognizing human posture was a key technology in video analysis. The OpenPose algorithm was one of the advantageous technologies that could accurately recognize multi-person poses in different environments in real time, so we used it innovatively to study consumer behavior in store. We hope to develop the potential of this application in the research of consumer behavior in store in the footwear retail industry by the technical advantages of the OpenPose algorithm. In our study, we first used an OpenPose algorithm to estimate multi-person pose and detection behavior, and then processed and recognized the videos collected in the store. We collected a week's surveillance video of a Red Dragonfly offline store from July 10 to July 16, 2020 in China. The specific process was to calibrate the area in the selected camera screen, then the algorithm performs identification and detection, and finally output in-store consumption Behavioral data. Our research results not only verified the feasibility of this application in offline retailing stores, but the data results also indicated that consumers tend to enter the store from the right, staying concentrated in the middle and back of the store. These results may be affected by the store space, product display, and staff guidance and reception.

Introduction

While online retailers enjoy a wealth of data on user clicks, hover times, and navigation paths, offline brick-and-mortar stores often operate in a data vacuum. Understanding how a customer moves through a store or which displays actually grab their attention usually requires expensive RFID tagging or labor-intensive manual observation.

This research addresses the challenge of "digitizing" the physical retail experience. The authors seek a non-intrusive, cost-effective way to automate the collection of behavioral data—such as foot traffic, dwell time, and product interaction—using existing infrastructure: the store's security cameras. By transforming standard surveillance footage into actionable insights, retailers can achieve data parity with their online counterparts.

Methodology

The researchers built a pipeline centered on OpenPose, a well-known "bottom-up" multi-person 2D pose estimation algorithm. Unlike "top-down" approaches that first detect a person and then find their joints, OpenPose detects all joint points (keypoints) in an image first and then associates them with individuals. This makes it particularly robust for crowded retail environments where shoppers often overlap.

Real-time multi-person pose estimation using OpenPose to detect consumer interactions.

The technical workflow included several sophisticated post-processing steps to handle real-world video noise:

  • Median Filtering: To account for "flicker" or missing data caused by temporary occlusions, a median filter was used to interpolate missing joint coordinates.
  • Digital FIR Low-Pass Filter: To remove high-frequency "jitter" in the skeletal tracking, a 17-order Finite Impulse Response (FIR) filter was applied, specifically tuned to the typical frequency of human walking (below 3Hz).
  • Behavioral Logic: "Intention to shop" was defined based on two triggers: standing in a calibrated 80cm "attention area" for more than 5 seconds, or the physical gesture of raising a hand to take a shoe from a shelf.

Results and Analysis

The study utilized a real-world dataset from a Red Dragonfly footwear store in Chengdu, China. The researchers manually calibrated five specific "attention areas" (labeled A through E) within the camera's field of view to serve as the ground truth for spatial analysis.

Summary of passenger flow and attention conversion rates across different store areas.

The system processed the week-long data to reveal high-level performance metrics:

  • Traffic Volume: The store saw a daily average of 1,765 passengers.
  • Engagement: Of those passengers, 1,302 showed "attention" behavior (dwelling or reaching), resulting in an overall conversion rate of 73.76%.
  • Spatial Efficiency: Area C emerged as the "hotspot," boasting a 41.2% conversion rate. In contrast, Areas A and E were almost entirely ignored (1.2%–2.2%).

These findings confirmed the "Right-Hand Bias" theory: consumers tend to enter a store and immediately veer to the right, with trajectories heavily concentrated on the right side and middle-rear of the store. This suggests that store layout and staff placement are more critical than total floor space.

Conclusion and Future Work

The research successfully demonstrates the feasibility of using low-cost, existing CCTV hardware combined with modern AI to produce high-quality behavioral data. While the single-camera setup faces challenges with occlusion in dense crowds, the results provide a clear path for retail optimization. Future work will focus on multi-camera fusion to eliminate blind spots and the integration of facial analysis to gather demographic data alongside behavioral intent.

Summary updated

Abstract

Due to the importance of offline consumer behavior, more and more people had begun to study consumer behavior in store. In offline consumer behavior research, the application of video analysis technology was the most direct and convenient. Recognizing human posture was a key technology in video analysis. The OpenPose algorithm was one of the advantageous technologies that could accurately recognize multi-person poses in different environments in real time, so we used it innovatively to study consumer behavior in store. We hope to develop the potential of this application in the research of consumer behavior in store in the footwear retail industry by the technical advantages of the OpenPose algorithm. In our study, we first used an OpenPose algorithm to estimate multi-person pose and detection behavior, and then processed and recognized the videos collected in the store. We collected a week's surveillance video of a Red Dragonfly offline store from July 10 to July 16, 2020 in China. The specific process was to calibrate the area in the selected camera screen, then the algorithm performs identification and detection, and finally output in-store consumption Behavioral data. Our research results not only verified the feasibility of this application in offline retailing stores, but the data results also indicated that consumers tend to enter the store from the right, staying concentrated in the middle and back of the store. These results may be affected by the store space, product display, and staff guidance and reception.
BibTeX
@misc{li2020application,
  author = {Jingjing Li and Jie Zeng and Keyu Hou and Jin Zhou and Rui Wang},
  title = {{Application of Openpose algorithm to detect consumer behavior in store}},
  year = {2020},
  doi = {10.24264/icams-2020.iii.11},
}
Figure