Introduction
Modern infrastructure is increasingly shifting from centralized to decentralized control architectures. While centralized systems offer high performance, they introduce critical vulnerabilities, such as single points of failure and susceptibility to network congestion or cyberattacks. However, decentralized control presents a significant challenge: enabling independent agents to coordinate on a globally optimal strategy without direct communication. This paper addresses this gap by proposing a reinforcement learning (RL) framework that allows multiple controllers to reach an $\epsilon$-Correlated Equilibrium (CE) using only local observations.
Decentralized Learning Framework
The authors frame the control problem as a Markov Game. To overcome the coordination barrier, they introduce two primary algorithmic components:
1. Regret Minimization for Correlated Equilibrium
Instead of seeking a Nash Equilibrium, which is often computationally intractable in decentralized settings, the framework targets a Correlated Equilibrium. In a CE, no agent has an incentive to deviate from a suggested strategy, provided others also follow it.
To achieve this without communication, each controller minimizes its "internal regret"—the difference between the cost incurred and the cost that would have been incurred had a different action been chosen consistently in the past. Since agents cannot observe counterfactual outcomes, they employ Importance Sampling to estimate these regrets. Over time, minimizing this regret sum leads the joint system to converge toward an $\epsilon$-CE.
2. Multi-objective Control via Blackwell’s Approachability
Real-world infrastructure management involves competing objectives, such as maintaining system stability (e.g., water levels) while minimizing energy consumption. The authors utilize Blackwell’s Approachability Theorem to dynamically adjust the weights of these objectives. This ensures that the system's long-term average cost vector stays within a predefined "safe zone" or target set.
To handle the discrete nature of physical controls (e.g., the number of active pumps) without the computational explosion typical of mixed-integer problems, the continuous state space (like reservoir levels) is partitioned into discrete zones.
Experimental Evaluation
The framework was evaluated using a simulation of a Danish District Metered Area (DMA). The simulation incorporates real-world water consumption data from a Danish utility and models the physical characteristics of Grundfos pumps and an elevated reservoir.
The system consists of two pumping stations (P1 and P2) that must coordinate to meet fluctuating demand while keeping the reservoir level within a stable range.
Results and Discussion
The proposed decentralized approach demonstrated both theoretical and empirical success:
- Convergence: The authors provide a formal proof that the decentralized RL algorithm converges to an $\epsilon$-Correlated Equilibrium almost surely under the unichain assumption.
- Empirical Stability: In simulations, the agents successfully stabilized the reservoir levels despite stochastic demand patterns.
- Regret Convergence: Analysis of the "Regret Sum" for the controllers showed a clear trend toward the negative orthant, confirming that the agents reached a stable equilibrium.
- Automated Weighting: The multi-objective weighting algorithm successfully discovered optimal weights for competing goals (e.g., balancing error vs. power) without requiring manual intervention, effectively balancing safety and efficiency.
Conclusion
This research provides a robust pathway for decentralized control in complex systems. By focusing on Correlated Equilibrium and leveraging regret minimization with Blackwell’s Approachability, the framework enables coordination without communication. Future work will explore the "curse of dimensionality" in finer state discretizations and the system's robustness against extreme sensor noise or external disturbances.
Summary updated