Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism

Yilin Liang, Yuping Hu, Luo Dong-jun, Qi Zhu, Qingxuan Chen, Chunmei WangSustainability 2023 · DOI

TL;DR

This paper tackles the challenge of optimizing dynamic pricing for e-retailers in a presale environment with uncertain demand and inventory backlogs. They propose a novel approach using Deep Reinforcement Learning, specifically a Dueling DQN algorithm, to learn optimal pricing strategies that maximize retailer profits.

What Problem Does This Paper Address?

In today's e-commerce landscape, retailers often use presale mechanisms to reduce manufacturing and ordering risks. However, setting the right price during this phase, especially for new products with unknown demand, is tricky. Traditional methods struggle with the dynamic nature of customer behavior, inventory levels, and fluctuating market conditions. This paper addresses the gap in optimizing real-time pricing strategies within these complex presale scenarios, considering factors like inventory backlog costs and the influence of past prices on consumer decisions.

Methods

The core of this paper is the application of Deep Reinforcement Learning (DRL) to solve the dynamic pricing problem. They model the sales environment as a Markov Decision Process (MDP), which allows an agent to learn optimal actions (pricing decisions) through trial and error.

Specifically, they employ a Dueling Deep Q-Network (Dueling DQN) algorithm. This is an advancement over standard DQN, which uses a dual-network structure to separate the estimation of state value and action advantages, leading to more accurate value function estimation and improved learning stability. The Dueling DQN-DP algorithm is designed to handle the multi-cycle pricing problem with a finite sales horizon, aiming to maximize cumulative retailer profits. They also compare their Dueling DQN approach against traditional tabular reinforcement learning algorithms like Q-learning and SARSA, as well as a standard DQN.

Datasets

The paper does not use real-world datasets. Instead, they create a simulated environment to train and evaluate their proposed Dueling DQN-DP algorithm. This simulation incorporates various factors such as consumer arrival rates (modeled as a Poisson distribution), demand functions that vary across presale, formal sales, and final sales phases, and inventory backlog costs. They also define four different market environments with varying parameters to test the algorithm's generalizability.

Key Insights and Ideas

  • Past Prices Matter: Consumers' purchasing decisions are significantly influenced by past prices, especially in the presale and early formal sales periods. Retailers need to account for this "reference price effect" when setting prices.
  • Inventory Backlog is Costly: High inventory backlog costs necessitate more aggressive pricing strategies, particularly deeper discounts in the early selling period, to move products and mitigate losses.
  • Blockchain for Transparency Boosts Demand: Integrating blockchain technology to enhance commodity traceability can increase consumer trust and willingness to purchase, ultimately boosting demand.
  • DRL Outperforms Traditional RL: Deep Reinforcement Learning, particularly the Dueling DQN approach, demonstrates superior performance and faster convergence compared to traditional tabular RL methods like Q-learning and SARSA in this dynamic pricing context.

Key Results

The Dueling DQN-DP algorithm achieved 91.58% of the optimal profit in their simulations, significantly outperforming Q-learning (77.84%) and SARSA (78.41%). The algorithm also showed faster convergence rates and greater stability compared to a standard DQN. Experiments across different market environments demonstrated the algorithm's universal applicability, consistently achieving over 90% of the optimal strategy's performance. The study also provided actionable insights for retailers, suggesting that they should adjust pricing based on the influence of past prices and inventory costs to maximize profits.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Monopolistic Pricing: The current study focuses on a single retailer. Future work should consider competitive environments with multiple retailers, where competitor pricing influences customer decisions.
  • Blockchain Transparency Risks: While blockchain enhances transparency, it also introduces potential data disclosure risks. Future research should explore more efficient smart contract implementations to mitigate these risks.
  • Consumer Behavior Assumptions: The model assumes consumers are loss-neutral. Future work could incorporate more complex consumer behavior models, considering factors like quality and maximizing expected utility across periods.
  • Risk-Sensitive Agents: The study could be extended to consider risk-sensitive sellers and consumers.
  • Return Rates: The impact of product return rates on revenue management should be investigated.

Summary updated

Abstract

Despite the emergence of a presale mechanism that reduces manufacturing and ordering risks for retailers, optimizing the real-time pricing strategy in this mechanism and unknown demand environment remains an unsolved issue. Consequently, we propose an automatic real-time pricing system for e-retailers under the inventory backlog impact in the presale mode, using deep reinforcement learning technology based on the Dueling DQN algorithm. This system models the multicycle pricing problem with a finite sales horizon as a Markov decision process (MDP) to cope with the uncertain environment. We train and evaluate the proposed environment and agent in a simulation environment and compare it with two tabular reinforcement learning algorithms (Q-learning and SARSA). The computational results demonstrate that our proposed real-time pricing learning framework for joint inventory impact can effectively maximize retailers’ profits and has universal applicability to a wide range of presale models. Furthermore, according to a series of experiments, we find that retailers should not neglect the impact of the presale or previous prices on consumers’ purchase behavior. If consumers pay more attention to past prices, the retailer must decrease the current price. When the cost of inventory backlog increases, they need to offer deeper discounts in the early selling period. Additionally, introducing blockchain technology can improve the transparency of commodity traceability information, thus increasing consumer demand for purchase.
BibTeX
@inproceedings{liang2023distributed,
  author = {Yilin Liang and Yuping Hu and Luo Dong-jun and Qi Zhu and Qingxuan Chen and Chunmei Wang},
  title = {{Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism}},
  booktitle = {Sustainability},
  year = {2023},
  doi = {10.3390/su151310480},
}
Figure