Reinforcement Learning for Inventory Control under Regime-Switching Demand: A Controlled Comparison of LSTM, MLP, and (s,S) Policies

Alex Haines, Nathanael L. BaisaarXiv 2026 · arXiv

TL;DR

Recurrent reinforcement learning (LSTM) significantly outperforms memoryless policies and classical heuristics for inventory control when demand fluctuates between hidden regimes. The advantage of recurrent memory scales dramatically with supply-chain lead time, as the agent learns to both infer latent demand states and track complex delivery pipelines.

What Problem Does This Paper Address?

Classical inventory theory often relies on the $(s,S)$ policy—ordering up to level $S$ whenever stock hits $s$. While provably optimal for stationary demand, this approach fails in the real world where demand is "regime-switching." A product might have a low baseline demand that suddenly spikes due to a promotion or a competitor's exit. These shifts are often hidden (latent) and unpredictable.

The challenge is compounded by lead time. If a manager misjudges a regime shift, the consequences (over-ordering or stockouts) are locked in for the duration of the replenishment pipeline. The authors investigate whether Deep Reinforcement Learning (DRL), specifically recurrent architectures like LSTMs, can solve this Partial Observability problem better than memoryless MLPs or classical heuristics.

Methods

The researchers developed a custom Gymnasium environment where demand is driven by a two-state Hidden Markov Model (HMM) with Poisson emissions (Low demand $\lambda=5$, High demand $\lambda=25$).

They compared three distinct policies:

  1. Optimized $(s,S)$: A classical baseline tuned via grid search for each lead time.
  2. MLP PPO: A memoryless "Multi-Layer Perceptron" agent using Proximal Policy Optimization. It sees a small 5-step window of demand history but cannot maintain long-term internal state.
  3. LSTM Recurrent PPO: An agent with a Long Short-Term Memory layer capable of accumulating evidence over many timesteps to infer the hidden demand regime.

To ensure a fair fight, the MLP was given extra layers so that its total parameter count matched the LSTM. Both agents were "warm-started" using behavioral cloning from an $(s,S)$ teacher to accelerate early training.

Datasets

The study uses a synthetic, controlled environment rather than a static dataset. This allowed the authors to know the "ground truth" regime during evaluation (to analyze agent behavior) while keeping it hidden from the agents during training. The demand process was designed with a mean persistence of 33 steps—long enough that a simple 5-step window (available to the MLP) is insufficient to reliably identify the current regime.

Key Insights and Ideas

  • The Lead Time Multiplier: Recurrence isn't just a marginal improvement; its value is tied directly to the "credit-assignment horizon." When lead time is zero, memoryless and recurrent policies perform almost identically. As lead time increases, the LSTM's ability to "remember" becomes the decisive factor.
  • Strategic Stockouts: The LSTM learns a sophisticated trade-off. It achieves higher rewards by accepting a lower service level (more stockouts) compared to the MLP, which defaults to a "defensive" (and expensive) over-stocking strategy because it cannot distinguish a transient spike from a regime shift.
  • Representational Shift: By applying Principal Component Analysis (PCA) to the LSTM's hidden states, the authors discovered a fascinating mechanical shift.
  • At short lead times, the hidden state acts as a binary regime indicator (detecting "High" vs "Low" demand).
  • At long lead times, the representation evolves into a "pipeline-tracking" state, managing the complexity of multiple orders currently in transit.
Fig. 2. PCA of the LSTM hidden state coloured by true demand regime. (Left) At lt = 0 the regimes form two separable clusters along PC1. (Right) At lt = 10 they interleave in a V-shape, indicating a shift from regime-indicator to pipeline-tracking encoding.

Key Results

  • Performance Gap: The LSTM outperformed all alternatives at every lead time. Its margin over the MLP grew monotonically: from a mere $1.01\times$ advantage at lead time 0 to a massive $2.17\times$ advantage at lead time 10.
  • Heuristic Failure: At the longest lead time ($lt=10$), the optimized $(s,S)$ policy actually became unprofitable (negative mean reward), whereas the LSTM remained highly profitable.
  • Statistical Robustness: Welch’s t-tests confirmed that the performance differences were statistically significant ($p < 0.001$) for all non-zero lead times.
Fig. 1. Mean evaluation reward versus lead time for the LSTM, MLP, and (s,S) policies, with one-standard-deviation (std) bands over 30 episodes.

Limitations and Future Work

  • Convergence: The authors noted that some training runs had not fully converged at 5 million steps, suggesting the performance gap might even widen with more compute.
  • Stylized Environment: The model uses a single-product, single-echelon setup with lost sales. Real-world supply chains are often multi-echelon networks where backorders are allowed.
  • HMM Complexity: The demand model was a simple two-state HMM. Future work could explore more complex transitions, non-Poisson emissions, or seasonality.
  • Baseline Comparison: A valuable next step would be comparing RL against a "regime-aware" $(s,S)$ policy that uses an offline HMM estimate to switch between different $(s,S)$ pairs.

Summary updated

Abstract

Classical inventory policies such as (s,S) are provably optimal under stationary demand but cannot adapt when demand switches between hidden regimes, and memoryless deep reinforcement learning policies, confined to a fixed observation window, cannot maintain a belief over latent demand structure. We ask whether a recurrent policy offers a measurable advantage over both a memoryless policy and a classical baseline under regime-switching demand, and how that advantage scales with supply-chain lead time. We build a custom Gymnasium environment in which daily demand is generated by a two-state hidden Markov process with Poisson emissions, so that the regime is known at evaluation time but hidden from the agent. An optimised (s,S) heuristic, an MLP PPO agent, and an LSTM Recurrent PPO agent are trained under matched compute and evaluated at lead times of 0, 3, 6, and 10 days over 30 deterministic episodes each. The recurrent policy outperforms both alternatives at every lead time, and its margin over the MLP widens monotonically with lead time, from parity at lead time 0 to 2.17$\times$ at lead time 10. Principal component analysis of the LSTM hidden state explains the widening gap: the representation shifts from encoding a binary regime indicator at short lead times to encoding multi-step pipeline state at long lead times. Recurrent memory becomes the decisive architectural factor as the credit-assignment horizon lengthens.
BibTeX
@misc{haines2026reinforcement,
  author = {Alex Haines and Nathanael L. Baisa},
  title = {{Reinforcement Learning for Inventory Control under Regime-Switching Demand: A Controlled Comparison of LSTM, MLP, and (s,S) Policies}},
  year = {2026},
  eprint = {2611.15945},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15945},
}
Figure