RetailOpt: Opt-In, Easy-to-Deploy Trajectory Estimation from Smartphone Motion Data and Retail Facility Information

Ryo Yonetani, Jun Baba, Yasutaka Furukawa

TL;DR

This paper introduces RetailOpt, a novel system for tracking customer movement in retail stores using only smartphone motion data and existing retail facility information (store maps, purchase records). It achieves this by combining inertial navigation with a continuous-discrete optimization framework, offering an opt-in, privacy-preserving, and cost-effective solution for understanding customer behavior.

What Problem Does This Paper Address?

Understanding customer movement within physical retail spaces is crucial for various applications like targeted marketing, in-store navigation, and inventory management. However, existing indoor localization and tracking systems often suffer from high costs due to extensive hardware installations (e.g., UWB systems, cameras) and maintenance requirements, or they necessitate expensive and time-consuming site surveys to collect environmental data (e.g., Wi-Fi fingerprints, visual features). Furthermore, privacy concerns are paramount, as many systems capture data without explicit user consent. This paper aims to bridge this gap by proposing a system that leverages readily available smartphone sensors and existing retail data, offering an opt-in solution that respects user privacy and minimizes infrastructure costs.

Methods

RetailOpt employs a two-stage optimization process:

  1. Continuous Optimization (Transformation Network):
    • Inertial Navigation: The system first uses smartphone Inertial Measurement Unit (IMU) data (accelerometer, gyroscope) to estimate a relative motion trajectory. This is a standard technique, but RetailOpt leverages a neural network (the "transformation network") to refine this initial trajectory.
  2. Transformation Network: This network is trained to transform the raw relative trajectory to better align with known locations (anchors) derived from retail facility information. It uses an attention mechanism to flexibly handle varying trajectory and anchor lengths. The objective function for this network includes a matching term (to align with anchors), a boundary constraint (to stay within the store's physical bounds), and a regularization term (to ensure smooth motion).
  3. Discrete Optimization (Viterbi Algorithm):
    • Graph Construction: After the continuous optimization, the trajectory is projected onto a 2D undirected graph. Vertices represent points on the transformed trajectory and randomly sampled points in the valid store space. Edges connect vertices if the line segment between them does not intersect obstacles.
  4. Viterbi Algorithm: This algorithm is used to find the lowest-cost path through the graph. The cost function encourages the final trajectory to stay close to the transformed trajectory while avoiding obstacles. This ensures the final estimated path is physically plausible within the store layout.

The key novelty lies in the fusion of relative inertial trajectories with sparse, time-unknown anchors (derived from purchase records and store maps) and the use of a continuous-discrete optimization pipeline to achieve accurate localization without extensive infrastructure.

Datasets

  • New Dataset: The authors collected data in two custom environments: a "store" (retail selling area with shelves) and an "office" (travel agency). This dataset involved 15 participants navigating these spaces, with ground truth trajectories obtained via visual-LiDAR SLAM.
  • Inertial Localization Dataset [12]: This public dataset was used for evaluation, comprising over 50 hours of IMU data from two large outdoor environments (University A, University B) and one indoor environment (Office). Ground truth was also obtained via SLAM.

Key Insights and Ideas

  • Leveraging Existing Data is Key: The core insight is that readily available retail facility information (store maps, purchase history) can serve as powerful, albeit sparse and time-ambiguous, anchors for localizing smartphone-based inertial trajectories. This bypasses the need for expensive dedicated localization hardware.
  • Opt-In for Privacy: The system is designed as an "opt-in" solution, empowering users to control their data, which is crucial for widespread adoption in privacy-sensitive environments like retail.
  • Bridging Continuous and Discrete: The combination of a learned continuous transformation (neural network) and a discrete optimization (Viterbi) effectively handles the challenges of noisy inertial data and sparse, time-ambiguous anchors.
  • Temporal Ambiguity is the Main Challenge: The paper highlights that the primary source of error in their system is often temporal discrepancy (not knowing when an anchor was visited) rather than spatial inaccuracy.

Key Results

  • Superior Accuracy: RetailOpt consistently outperformed baseline methods (TSP Solver, RONIN, various CPD variants) in terms of average positional error across both their new dataset and the public Inertial Localization Dataset.
  • Significant Error Reduction: On their new dataset, RetailOpt achieved average positional errors as low as 0.47m (store) and 0.83m (office) depending on how the smartphone was held. On the Inertial Localization Dataset, errors were around 1.35-2.62m for distances up to 400m.
  • Efficiency: The system can process trajectories of approximately 600 seconds in less than 2 seconds on a single GPU (NVIDIA Tesla T4).
  • No Site Survey Required: Unlike methods like NILoc [12], RetailOpt does not require training data specific to each environment, making it easier to deploy in new locations.

Limitations and Future Work

  • Item-to-Shelf Assumption: The system assumes a one-to-one mapping between items and their locations. If identical items are on multiple shelves, this could create ambiguity.
  • Single Store Data: The current approach assumes store maps and purchase records come from a single store. Extending this to multi-store scenarios would require a unified data platform and careful consideration of data protection and incentive design.
  • Smartphone Pose Variability: Performance can degrade when the smartphone's pose is highly variable (e.g., loosely in a bag), as this affects the accuracy of the initial inertial trajectory. The authors suggest that using IMUs from more rigidly attached devices (like watches) could improve stability.
  • Future Work: The authors suggest applying RetailOpt to other domains beyond retail, such as office maintenance or museum visits, where similar facility information and user interaction data might be available. They also point to the need for appropriate incentive designs and data protection frameworks for multi-store applications.

Summary updated

BibTeX
@inproceedings{yonetani2024retailopt,
  author = {Ryo Yonetani and Jun Baba and Yasutaka Furukawa},
  title = {{RetailOpt: Opt-In, Easy-to-Deploy Trajectory Estimation from Smartphone Motion Data and Retail Facility Information}},
  booktitle = {ISWC},
  year = {2024},
  doi = {10.1145/3675095.3676623},
}
Figure