Abstract
We study the problem of allocating impressions to sellers in e-commerce websites, aiming to maximize total revenue while accounting for the strategic behavior of sellers. We model this as a Markov decision process and propose IA(GRU), an actor-critic policy gradient algorithm based on DDPG. IA(GRU) addresses high-dimensionality and permutation invariance through a novel architecture using Gated Recurrent Units (GRUs) and sub-actor networks. Our evaluation against standard DDPG and heuristics shows that IA(GRU) achieves superior convergence, stability, and total revenue across various seller rationality models.
Introduction
In major e-commerce platforms like Amazon or Taobao, the mechanism for impression allocation—deciding which sellers to display to a buyer—is a primary driver of revenue. Traditional allocation methods often rely on static heuristics or historical scores. However, these methods fail to account for the fact that sellers are strategic agents who adjust their prices to manipulate their future visibility. A seller might lower prices today to boost their "historical score," gaining more impressions and profit tomorrow. This strategic interplay necessitates a mechanism design approach that is robust to long-term seller behavior and scalable to thousands of participants.
Methodology: The IA(GRU) Framework
The core of our approach is the IA(GRU) algorithm, which frames impression allocation as a Reinforcement Mechanism Design problem. Building upon the Deep Deterministic Policy Gradient (DDPG) framework, IA(GRU) introduces several architectural innovations to handle the complexities of e-commerce environments.
Permutation Invariance and Scalability
Standard neural networks are sensitive to the ordering of inputs. In e-commerce, treating "Seller A" and "Seller B" as fixed indices is inefficient, as their relative positions shouldn't dictate the allocation logic. IA(GRU) achieves permutation invariance by using a shared background network and GRUs to process seller features and history. This ensures the model learns to respond to seller behaviors (e.g., pricing trends, transaction rates) rather than specific identities.
Furthermore, the architecture employs sub-actor and sub-critic networks for each seller. This modular design allows the system to handle a variable number of sellers—a common occurrence in real-world markets where participants enter and leave—without requiring retraining.
Modeling Strategic Sellers
To train the agent, we simulate sellers using "no-regret" learning models, including $\epsilon$-Greedy, UCB1, and Exp3. This assumes sellers are not perfectly rational but are learners attempting to optimize their own profits over time. By "playing" against these simulated learners, the IA(GRU) agent learns an allocation policy that nudges the ecosystem toward higher platform revenue.
Experimental Results
We evaluated IA(GRU) in simulated environments mimicking Taobao dynamics, scaling from 200 to 10,000 sellers.
Convergence and Stability
A significant challenge in applying standard RL to multi-agent mechanism design is stability. In our experiments, standard DDPG often failed to converge, with its loss functions oscillating significantly. In contrast, IA(GRU) demonstrated stable convergence.
As shown in the reward plots, IA(GRU) effectively learns a policy that increases total platform reward (revenue) over time, whereas DDPG struggles to find a stable, improving strategy.
Revenue Performance
IA(GRU) consistently outperformed common heuristics, such as Greedy Myopic (allocating based solely on the previous round's revenue) and Linear UCB (a contextual bandit approach). The algorithm maintained its performance lead across all tested seller rationality models and proved robust in "variable seller" settings where production costs fluctuated each round.
Conclusion
IA(GRU) represents a step forward in applying deep reinforcement learning to large-scale mechanism design. By incorporating permutation invariance and modular sub-networks, the framework overcomes the limitations of standard RL in high-dimensional, dynamic environments. Future work will explore heterogeneous product settings and more complex human-like seller behaviors to further bridge the gap between simulation and real-world deployment.
Summary updated