Closing the Loop: Coordinating Inventory and Recommendation via Deep Reinforcement Learning on Multiple Timescales

Jinyang Jiang, Jinhui Han, Yijie Peng, Ying ZhangarXiv 2025 · DOI · arXiv

Introduction

In modern e-commerce, operational efficiency and marketing effectiveness are often treated as independent objectives. Operations teams manage inventory replenishment to minimize stockouts and holding costs, while marketing departments optimize recommendation algorithms to maximize customer engagement and sales. However, this siloed approach frequently leads to sub-optimal outcomes, such as promoting out-of-stock items or overstocking products with low demand.

This paper introduces a Multi-Agent Reinforcement Learning (MARL) framework designed to "close the loop" between these two functions. By coordinating inventory and recommendation decisions, the system aims to achieve higher system-wide profitability and operational stability.

Problem Formulation and Coordination Mechanisms

The core challenge lies in the dynamic and non-linear interactions between supply and demand. The researchers identify two primary coordination mechanisms:

  1. Demand Smoothing: Adjusting recommendations to stabilize demand based on current inventory levels.
  2. Adaptive Ordering: Proactively adjusting replenishment orders in anticipation of marketing-driven demand shifts.
Timeline of events and decision-making within a single period $t$.

As shown in the figure above, the system operates over discrete time periods. At the start of period $t$, shipments ordered previously arrive, and the current inventory status is assessed. Decisions on replenishment ($q_t^i$) and recommendations ($\alpha_t^{i,j}$) are made, which subsequently influence customer demand ($D_t^i$) and the resulting backlog ($U_t^i$) or ending inventory.

Methodology: Multi-Timescale Multi-Agent RL (MTMA)

The proposed framework decomposes the complex joint optimization problem into two specialized agents: an inventory agent ($\pi_q$) and a recommendation agent ($\pi_\alpha$). To address the "curse of dimensionality" and the inherent differences in task complexity, the authors employ a Multi-Timescale Learning approach.

Inspired by the concept that different cognitive tasks require different learning rates, the framework assigns:

  • Fast Timescale (Inventory): Inventory decisions are structurally more stable and are updated with larger step sizes.
  • Slow Timescale (Recommendation): Recommendation policies are high-dimensional and sensitive, requiring smaller step sizes to ensure convergence and stability.

This asymmetric learning allows the simpler inventory agent to effectively track the conditional optimum of the more complex recommendation agent, leading to faster system-wide stabilization.

Experimental Results

The framework was evaluated through extensive simulations involving multiple products and customers under various demand distributions (e.g., Poisson, Bernoulli).

Profitability and Efficiency

The Cooperative (MTMA) approach significantly outperformed siloed benchmarks. In primary simulations, it achieved an average total profit of 584.32, compared to just 157.82 for isolated decision-making. This improvement stems from the agents' ability to internalize substitution effects—learning to reduce recommendations for one product when pushing another to manage limited inventory.

Stability and Convergence

The multi-timescale design demonstrated superior stability, characterized by narrower confidence intervals and faster convergence compared to single-timescale baselines. By ensuring that "pushed" products were available in the warehouse, the system drastically reduced inventory costs related to holding and stockout penalties.

Conclusion and Future Directions

The MTMA framework successfully demonstrates the value of cross-functional coordination in e-commerce. While the current study focused on a controlled number of products, future work will explore hierarchical decomposition to scale the system to millions of SKUs. Additionally, extending the model to include multi-tier coordination with upstream suppliers could further enhance supply chain resilience.

Citation: Jiang, J., Han, J., Peng, Y., & Zhang, Y. (2025). Closing the Loop: Coordinating Inventory and Recommendation via Deep Reinforcement Learning on Multiple Timescales. arXiv:2510.04272v1 [cs.AI].

Summary updated

Abstract

Effective cross-functional coordination is essential for enhancing firm-wide profitability, particularly in the face of growing organizational complexity and scale. Recent advances in artificial intelligence, especially in reinforcement learning (RL), offer promising avenues to address this fundamental challenge. This paper proposes a unified multi-agent RL framework tailored for joint optimization across distinct functional modules, exemplified via coordinating inventory replenishment and personalized product recommendation. We first develop an integrated theoretical model to capture the intricate interplay between these functions and derive analytical benchmarks that characterize optimal coordination. The analysis reveals synchronized adjustment patterns across products and over time, highlighting the importance of coordinated decision-making. Leveraging these insights, we design a novel multi-timescale multi-agent RL architecture that decomposes policy components according to departmental functions and assigns distinct learning speeds based on task complexity and responsiveness. Our model-free multi-agent design improves scalability and deployment flexibility, while multi-timescale updates enhance convergence stability and adaptability across heterogeneous decisions. We further establish the asymptotic convergence of the proposed algorithm. Extensive simulation experiments demonstrate that the proposed approach significantly improves profitability relative to siloed decision-making frameworks, while the behaviors of the trained RL agents align closely with the managerial insights from our theoretical model. Taken together, this work provides a scalable, interpretable RL-based solution to enable effective cross-functional coordination in complex business settings.
BibTeX
@misc{jiang2025closing,
  author = {Jinyang Jiang and Jinhui Han and Yijie Peng and Ying Zhang},
  title = {{Closing the Loop: Coordinating Inventory and Recommendation via Deep Reinforcement Learning on Multiple Timescales}},
  year = {2025},
  eprint = {2510.04272},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.04272},
}
Figure