Reinforcement Mechanism Design for e-commerce

Qingpeng Cai, Aris Filos-Ratsikas, Pingzhong Tang, Yiwei Zhang

Abstract

We study the problem of allocating impressions to sellers in e-commerce websites, aiming to maximize total revenue while accounting for the strategic behavior of sellers. We model this as a Markov decision process and propose IA(GRU), an actor-critic policy gradient algorithm based on DDPG. IA(GRU) addresses high-dimensionality and permutation invariance through a novel architecture using Gated Recurrent Units (GRUs) and sub-actor networks. Our evaluation against standard DDPG and heuristics shows that IA(GRU) achieves superior convergence, stability, and total revenue across various seller rationality models.

Introduction

In major e-commerce platforms like Amazon or Taobao, the mechanism for impression allocation—deciding which sellers to display to a buyer—is a primary driver of revenue. Traditional allocation methods often rely on static heuristics or historical scores. However, these methods fail to account for the fact that sellers are strategic agents who adjust their prices to manipulate their future visibility. A seller might lower prices today to boost their "historical score," gaining more impressions and profit tomorrow. This strategic interplay necessitates a mechanism design approach that is robust to long-term seller behavior and scalable to thousands of participants.

Methodology: The IA(GRU) Framework

The core of our approach is the IA(GRU) algorithm, which frames impression allocation as a Reinforcement Mechanism Design problem. Building upon the Deep Deterministic Policy Gradient (DDPG) framework, IA(GRU) introduces several architectural innovations to handle the complexities of e-commerce environments.

The IA(GRU) architecture featuring permutation-invariant GRU layers and sub-actor modules.

Permutation Invariance and Scalability

Standard neural networks are sensitive to the ordering of inputs. In e-commerce, treating "Seller A" and "Seller B" as fixed indices is inefficient, as their relative positions shouldn't dictate the allocation logic. IA(GRU) achieves permutation invariance by using a shared background network and GRUs to process seller features and history. This ensures the model learns to respond to seller behaviors (e.g., pricing trends, transaction rates) rather than specific identities.

Furthermore, the architecture employs sub-actor and sub-critic networks for each seller. This modular design allows the system to handle a variable number of sellers—a common occurrence in real-world markets where participants enter and leave—without requiring retraining.

Modeling Strategic Sellers

To train the agent, we simulate sellers using "no-regret" learning models, including $\epsilon$-Greedy, UCB1, and Exp3. This assumes sellers are not perfectly rational but are learners attempting to optimize their own profits over time. By "playing" against these simulated learners, the IA(GRU) agent learns an allocation policy that nudges the ecosystem toward higher platform revenue.

Experimental Results

We evaluated IA(GRU) in simulated environments mimicking Taobao dynamics, scaling from 200 to 10,000 sellers.

Convergence and Stability

A significant challenge in applying standard RL to multi-agent mechanism design is stability. In our experiments, standard DDPG often failed to converge, with its loss functions oscillating significantly. In contrast, IA(GRU) demonstrated stable convergence.

Comparison of training rewards: DDPG (left) shows high variance and lack of improvement, while IA(GRU) (right) shows clear upward progression and stabilization.

As shown in the reward plots, IA(GRU) effectively learns a policy that increases total platform reward (revenue) over time, whereas DDPG struggles to find a stable, improving strategy.

Revenue Performance

IA(GRU) consistently outperformed common heuristics, such as Greedy Myopic (allocating based solely on the previous round's revenue) and Linear UCB (a contextual bandit approach). The algorithm maintained its performance lead across all tested seller rationality models and proved robust in "variable seller" settings where production costs fluctuated each round.

Conclusion

IA(GRU) represents a step forward in applying deep reinforcement learning to large-scale mechanism design. By incorporating permutation invariance and modular sub-networks, the framework overcomes the limitations of standard RL in high-dimensional, dynamic environments. Future work will explore heterogeneous product settings and more complex human-like seller behaviors to further bridge the gap between simulation and real-world deployment.

Summary updated

Abstract

We study the problem of allocating impressions to sellers in e-commerce websites, such as Amazon, eBay or Taobao, aiming to maximize the total revenue generated by the platform. We employ a general framework of reinforcement mechanism design, which uses deep reinforcement learning to design efficient algorithms, taking the strategic behaviour of the sellers into account. Specifically, we model the impression allocation problem as a Markov decision process, where the states encode the history of impressions, prices, transactions and generated revenue and the actions are the possible impression allocations in each round. To tackle the problem of continuity and high-dimensionality of states and actions, we adopt the ideas of the DDPG algorithm to design an actor-critic policy gradient algorithm which takes advantage of the problem domain in order to achieve convergence and stability. We evaluate our proposed algorithm, coined IA(GRU), by comparing it against DDPG, as well as several natural heuristics, under different rationality models for the sellers - we assume that sellers follow well-known no-regret type strategies which may vary in their degree of sophistication. We find that IA(GRU) outperforms all algorithms in terms of the total revenue.
BibTeX
@inproceedings{cai2018reinforcement,
  author = {Qingpeng Cai and Aris Filos-Ratsikas and Pingzhong Tang and Yiwei Zhang},
  title = {{Reinforcement Mechanism Design for e-commerce}},
  booktitle = {The Web Conference},
  year = {2018},
  doi = {10.1145/3178876.3186039},
}
Figure