A deep reinforcement learning framework for allocating buyer impressions in e-commerce websites

Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, Yiwei Zhang

Abstract

We study the problem of allocating impressions to sellers in e-commerce websites, such as Amazon, eBay or Taobao, aiming to maximize the total revenue generated by the platform. We employ a general framework of reinforcement mechanism design, which uses deep reinforcement learning to design efficient algorithms, taking the strategic behaviour of the sellers into account. Specifically, we model the impression allocation problem as a Markov decision process, where the states encode the history of impressions, prices, transactions and generated revenue and the actions are the possible impression allocations in each round. To tackle the problem of continuity and high-dimensionality of states and actions, we adopt the ideas of the DDPG algorithm to design an actor-critic policy gradient algorithm which takes advantage of the problem domain in order to achieve convergence and stability. We evaluate our proposed algorithm, coined IA(GRU), by comparing it against DDPG, as well as several natural heuristics, under different rationality models for the sellers. We find that IA(GRU) outperforms all algorithms in terms of the total revenue.

Introduction

In modern e-commerce, "impression allocation"—deciding which products to show a buyer and in what order—is a critical driver of platform revenue. Traditional approaches often rely on simple heuristics or historical performance scores. However, these methods frequently ignore the strategic nature of sellers, who dynamically adjust their prices based on the visibility they receive.

This paper addresses the challenge of strategic complexity and high dimensionality in market states. Sellers are not static; they employ learning algorithms to maximize their own profits, creating a moving target for the platform. We propose IA(GRU), a deep reinforcement learning (DRL) framework that treats impression allocation as a mechanism design problem, optimizing platform revenue against sophisticated, "no-regret" seller strategies.

Methodology: The IA(GRU) Architecture

The core of our approach is an actor-critic policy gradient algorithm designed to handle the unique constraints of e-commerce environments. Standard neural networks often struggle with permutation invariance—treating the "first seller" differently from the "second" simply due to their position in an input list.

The IA(GRU) architecture utilizes a Gated Recurrent Unit (GRU) and sub-actor networks to achieve permutation invariance and scalability.

As shown in the architecture diagram, IA(GRU) incorporates several key innovations:

  1. Permutation Invariance: By using a GRU and a sorting mechanism, the model ensures that sellers are evaluated based on performance data rather than arbitrary IDs.
  2. Sub-actor Networks: To manage thousands of sellers, the "Actor" network is decomposed into sub-actors that process individual seller features, which are then aggregated to form the final allocation.
  3. Scale-and-Solve: For real-world deployment, the large-scale allocation problem is divided into parallelizable sub-problems.

Experimental Evaluation

We evaluated IA(GRU) in a simulated environment where sellers follow four distinct "no-regret" rationality models: $\epsilon$-Greedy, $\epsilon$-First, UCB1, and Exp3. These models represent varying degrees of seller sophistication. Buyer valuations were drawn from a uniform distribution $U(0,1)$. The experiments scaled from a base of 200 sellers up to 10,000 to test computational efficiency and robustness.

Results and Stability Analysis

A primary finding is the superior stability of IA(GRU) compared to standard DRL approaches like Deep Deterministic Policy Gradient (DDPG). While vanilla DDPG attempts to learn a global coordination strategy that becomes exponentially complex as the number of sellers grows, IA(GRU)’s localized sub-network approach remains effective.

Comparison of training loss: DDPG (left) exhibits instability and failure to converge, while IA(GRU) (right) shows steady convergence.

Key performance metrics include:

  • Convergence: IA(GRU) demonstrated stable training curves, whereas DDPG showed high variance and failed to settle on a stable policy.
  • Revenue Maximization: IA(GRU) consistently outperformed benchmarks, including Greedy Myopic algorithms and Linear UCB, across all seller rationality models.
  • Robustness: The framework maintained its advantage even in heterogeneous pools where sellers utilized different pricing strategies simultaneously.

Conclusion and Future Work

While IA(GRU) shows significant promise, it currently assumes that the platform has access to historical price and transaction data. Future work could explore scenarios with noisy or incomplete information. Additionally, the "Reinforcement Mechanism Design" framework introduced here could be extended to other domains involving multi-agent interactions, such as sponsored search auctions or dynamic pricing in ride-sharing applications.

Summary updated

Abstract

We study the problem of allocating impressions to sellers in e-commerce websites, such as Amazon, eBay or Taobao, aiming to maximize the total revenue generated by the platform. We employ a general framework of reinforcement mechanism design, which uses deep reinforcement learning to design efficient algorithms, taking the strategic behaviour of the sellers into account. Specifically, we model the impression allocation problem as a Markov decision process, where the states encode the history of impressions, prices, transactions and generated revenue and the actions are the possible impression allocations in each round. To tackle the problem of continuity and high-dimensionality of states and actions, we adopt the ideas of the DDPG algorithm to design an actor-critic policy gradient algorithm which takes advantage of the problem domain in order to achieve convergence and stability. We evaluate our proposed algorithm, coined IA(GRU), by comparing it against DDPG, as well as several natural heuristics, under different rationality models for the sellers - we assume that sellers follow well-known no-regret type strategies which may vary in their degree of sophistication. We find that IA(GRU) outperforms all algorithms in terms of the total revenue.
BibTeX
@inproceedings{cai2018deep,
  author = {Qingpeng Cai and Aris Filos-Ratsikas and Pingzhong Tang and Yiwei Zhang},
  title = {{A deep reinforcement learning framework for allocating buyer impressions in e-commerce websites}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3178876.3186039},
}
Figure