MIRACL: A Diverse Meta-Reinforcement Learning for Multi-Objective Multi-Echelon Combinatorial Supply Chain Optimisation

Rifny Rachman, Josh Tingey, Richard Allmendinger, Pan, Wei, Ph. D. Massachusetts Institute of Technology, Pradyumn Shukla, Bahrul Ilmi Nasution

TL;DR

MIRACL is a new meta-reinforcement learning framework designed to solve complex supply chain problems with multiple conflicting goals (like maximizing profit while minimizing emissions). By using a hierarchical approach and a diversity-seeking "simulated annealing" mechanism, it allows AI agents to adapt to new supply chain configurations in just a few steps, outperforming traditional methods in both efficiency and the quality of trade-off solutions.

What Problem Does This Paper Address?

Managing a modern supply chain is a "combinatorial" nightmare. You have multiple echelons (suppliers, factories, warehouses, retailers), high uncertainty in demand, and conflicting objectives—you want to maximize profit, but you also need to keep service levels high and carbon emissions low.

While Multi-Objective Reinforcement Learning (MORL) has shown promise in finding these trade-offs, it has a major "retraining bottleneck." If a shipping route is disrupted or a new warehouse opens, a standard MORL agent usually needs to be retrained from scratch, which is computationally expensive and slow. The challenge is creating an agent that can "learn how to learn," allowing it to generalize across different supply chain tasks and adapt to new environments with minimal data (few-shot learning).

Methods

The authors introduce MIRACL (Meta multI-objective Reinforcement leArning with Composite Learning), which builds on the Model-Agnostic Meta-Learning (MAML) framework. It introduces two primary innovations:

  1. Hierarchical Composite Learning: Instead of training on one task with one set of preferences at a time, MIRACL decomposes a single supply chain task into multiple "subproblems" with different weightings for the objectives (e.g., one subproblem might prioritize profit, another might prioritize green energy). Because these subproblems share the same underlying environment dynamics, the training signal is much more stable and less "noisy" than standard meta-learning.
  2. Archive-Guided Pareto Simulated Annealing (PSA): To ensure the agent doesn't just find one "easy" solution, MIRACL maintains an archive of the best trade-offs found so far. It uses a simulated annealing rule to actively nudge the preference weights toward unexplored regions of the objective space. This "pushes" the agent to discover a wider, more diverse range of solutions (the Pareto Front).

The framework uses Proximal Policy Optimization (PPO) as the underlying engine for the reinforcement learning agents.

Datasets

The researchers evaluated MIRACL using two distinct setups:

  • Messiah SC Simulator: A specialized supply chain simulator. They tested three levels of complexity:
  • Simple: 7 nodes, 8 edges (2 markets).
  • Moderate: 13 nodes, 21 edges (3 markets).
  • Complex: 24 nodes, 59 edges (5 markets).
  • MO-Gymnasium Benchmarks: To prove the method is "domain-agnostic" (not just for supply chains), they tested it on standard multi-objective robotics tasks: mo-hopper-v4, mo-halfcheetah-v4, and resource-gathering-v0.

Key Insights and Ideas

  • Stability through Shared Dynamics: By conditioning multiple subproblems on a single task within a meta-iteration, MIRACL significantly reduces gradient variance. This makes the "meta-update" (the global learning step) much more reliable.
  • Active vs. Passive Diversity: Most meta-learning algorithms sample preferences randomly. MIRACL shows that actively perturbing weights based on a historical archive leads to much better coverage of the trade-off space.
  • Operational "Smoothness": A fascinating qualitative finding is that MIRACL produces more stable production and inventory profiles over time. Traditional RL often results in "jittery" or erratic behavior, whereas MIRACL’s meta-policy prioritizes consistent, generalized strategies that are more practical for real-world manufacturing.

Key Results

  • Performance Gains: In simple to moderate tasks, MIRACL achieved up to 10% higher hypervolume (a measure of the quality and diversity of the trade-off set) and 5% better expected utility compared to standard Meta-MORL.
  • Efficiency: MIRACL adapted to new tasks in just 5,000 to 20,000 steps, whereas traditional RL baselines typically require around 500,000 steps to reach similar performance from scratch.
  • Generalization: On the mo-hopper-v4 robotics benchmark, MIRACL significantly outperformed the state-of-the-art Meta-MORL baseline across all metrics, proving its utility beyond logistics.

Limitations and Future Work

The authors note that as the supply chain grows to "Complex" levels (24+ nodes), the advantage of meta-learning over training-from-scratch (MORL/D) narrows. In these high-dimensional environments, the "few-shot" constraint becomes very difficult to satisfy because the search space is so vast.

Future work will likely focus on:

  • Scaling to Massive Networks: Improving exploration strategies to handle even larger, more realistic global supply chains.
  • Hybrid Meta-Learning: Combining different meta-policy training methods to balance rapid adaptation with deeper exploration.
  • Non-Linear Scalarization: Moving beyond linear weights to better capture non-convex trade-off regions.

Summary updated

Abstract

Multi-objective reinforcement learning (MORL) is effective for multi-echelon combinatorial supply chain optimisation, where tasks involve high dimensionality, uncertainty, and competing objectives. However, its deployment in dynamic environments is hindered by the need for task-specific retraining and substantial computational cost. We introduce MIRACL (Meta multI-objective Reinforcement leArning with Composite Learning), a hierarchical Meta-MORL framework that allows for a few-shot generalisation across diverse tasks. MIRACL decomposes each task into structured subproblems for efficient policy adaptation and meta-learns a global policy across tasks using a Pareto-based adaptation strategy to encourage diversity in meta-training and fine-tuning. To our knowledge, this is the first integration of Meta-MORL with such mechanisms in combinatorial optimisation. Although validated in the supply chain domain, MIRACL is theoretically domain-agnostic and applicable to broader dynamic multi-objective decision-making problems. Empirical evaluations show that MIRACL outperforms conventional MORL baselines in simple to moderate tasks, achieving up to 10% higher hypervolume and 5% better expected utility. These results underscore the potential of MIRACL for robust, efficient adaptation in multi-objective problems.
BibTeX
@misc{rachman2026miracl,
  author = {Rifny Rachman and Josh Tingey and Richard Allmendinger and Pan, Wei, Ph. D. Massachusetts Institute of Technology and Pradyumn Shukla and Bahrul Ilmi Nasution},
  title = {{MIRACL: A Diverse Meta-Reinforcement Learning for Multi-Objective Multi-Echelon Combinatorial Supply Chain Optimisation}},
  year = {2026},
  eprint = {2603.05760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05760},
}
Figure