Introduction
In modern e-commerce, operational efficiency and marketing effectiveness are often treated as independent objectives. Operations teams manage inventory replenishment to minimize stockouts and holding costs, while marketing departments optimize recommendation algorithms to maximize customer engagement and sales. However, this siloed approach frequently leads to sub-optimal outcomes, such as promoting out-of-stock items or overstocking products with low demand.
This paper introduces a Multi-Agent Reinforcement Learning (MARL) framework designed to "close the loop" between these two functions. By coordinating inventory and recommendation decisions, the system aims to achieve higher system-wide profitability and operational stability.
Problem Formulation and Coordination Mechanisms
The core challenge lies in the dynamic and non-linear interactions between supply and demand. The researchers identify two primary coordination mechanisms:
- Demand Smoothing: Adjusting recommendations to stabilize demand based on current inventory levels.
- Adaptive Ordering: Proactively adjusting replenishment orders in anticipation of marketing-driven demand shifts.
As shown in the figure above, the system operates over discrete time periods. At the start of period $t$, shipments ordered previously arrive, and the current inventory status is assessed. Decisions on replenishment ($q_t^i$) and recommendations ($\alpha_t^{i,j}$) are made, which subsequently influence customer demand ($D_t^i$) and the resulting backlog ($U_t^i$) or ending inventory.
Methodology: Multi-Timescale Multi-Agent RL (MTMA)
The proposed framework decomposes the complex joint optimization problem into two specialized agents: an inventory agent ($\pi_q$) and a recommendation agent ($\pi_\alpha$). To address the "curse of dimensionality" and the inherent differences in task complexity, the authors employ a Multi-Timescale Learning approach.
Inspired by the concept that different cognitive tasks require different learning rates, the framework assigns:
- Fast Timescale (Inventory): Inventory decisions are structurally more stable and are updated with larger step sizes.
- Slow Timescale (Recommendation): Recommendation policies are high-dimensional and sensitive, requiring smaller step sizes to ensure convergence and stability.
This asymmetric learning allows the simpler inventory agent to effectively track the conditional optimum of the more complex recommendation agent, leading to faster system-wide stabilization.
Experimental Results
The framework was evaluated through extensive simulations involving multiple products and customers under various demand distributions (e.g., Poisson, Bernoulli).
Profitability and Efficiency
The Cooperative (MTMA) approach significantly outperformed siloed benchmarks. In primary simulations, it achieved an average total profit of 584.32, compared to just 157.82 for isolated decision-making. This improvement stems from the agents' ability to internalize substitution effects—learning to reduce recommendations for one product when pushing another to manage limited inventory.
Stability and Convergence
The multi-timescale design demonstrated superior stability, characterized by narrower confidence intervals and faster convergence compared to single-timescale baselines. By ensuring that "pushed" products were available in the warehouse, the system drastically reduced inventory costs related to holding and stockout penalties.
Conclusion and Future Directions
The MTMA framework successfully demonstrates the value of cross-functional coordination in e-commerce. While the current study focused on a controlled number of products, future work will explore hierarchical decomposition to scale the system to millions of SKUs. Additionally, extending the model to include multi-tier coordination with upstream suppliers could further enhance supply chain resilience.
Citation: Jiang, J., Han, J., Peng, Y., & Zhang, Y. (2025). Closing the Loop: Coordinating Inventory and Recommendation via Deep Reinforcement Learning on Multiple Timescales. arXiv:2510.04272v1 [cs.AI].
Summary updated