Introduction
Supply Chain Management (SCM) is inherently volatile, frequently disrupted by "emergent" demand changes—sudden spikes or drops that deviate from planned quantities. While Reinforcement Learning (RL) has shown promise in optimizing SCM, most existing research assumes a centralized environment where a single controller manages the entire chain. In reality, supply chains are composed of independent entities that must coordinate to reach agreements. Static policies often fail during sudden shifts because they lack a mechanism for decentralized, real-time plan adjustment.
The Proposed Framework
The authors introduce an automated negotiation framework that allows independent RL agents to adapt to sudden demand fluctuations. The model utilizes a four-layer serial supply chain consisting of a Terminal Supplier (TS), a Supplier (S), a Customer (C), and a Terminal Customer (TC).
Key Innovations
- Value-Based Utility: Instead of using hard-coded utility functions, agents use their RL value function ($V$) to estimate the "worth" of a proposed plan. By running internal simulations, an agent uses its learned experience to judge whether a new agreement is beneficial.
- Negotiation Trigger: Agents initiate negotiation only when the difference between their current state's value and the potential value of a new plan exceeds a specific threshold ($\tau$). This allows for early detection of terminal demand changes before inventory shortages occur.
- Alternating Offers Protocol (AOP): Agents trade offers back and forth to reach a consensus, allowing for iterative refinement of plans during complex, evolving crises.
Experimental Results
The researchers evaluated their PrePV-Nego framework against a "Zero-Shot" RL agent (no negotiation) and a standard "Base-Stock" heuristic across four demand scenarios. The primary performance metric was the Achievement Rate (AR), which measures the percentage of steps where inventory was sufficient to meet demand.
As shown in the results table, PrePV-Nego achieved a 1.00 (100%) achievement rate across all scenarios, including the most challenging "Increase (large)" scenario. In contrast, the Zero-Shot agent's performance plummeted to 0.43 for the Customer agent, and the Base-Stock model dropped to 0.67. The negotiation process successfully "convinced" the chain to move more product, resulting in shipping quantities 33% higher and inventory levels 52% higher than the non-negotiating models.
Conclusion and Future Work
By leveraging the RL value function as a utility metric, this framework bridges the gap between long-term strategic planning and short-term tactical adjustment. While highly effective in a serial chain, future research is needed to address the computational costs of internal simulations and the scalability of negotiation overhead in more complex, non-linear supply networks. Additionally, exploring scenarios where agents have conflicting, rather than cooperative, incentives remains a key area for development.
Summary updated