Multi-Agent Learning for Resilient Distributed Control Systems

Yuhan Zhao, Craig Rieger, Quanyan ZhuarXiv 2022 · DOI · arXiv

Introduction

As critical infrastructure—including power grids and transportation networks—becomes increasingly digitized, the limitations of traditional security models are becoming apparent. This research proposes a shift from "perfect security" toward "resilience": the ability of Industrial Control Systems (ICS) to autonomously detect, adapt to, and recover from sophisticated cyber-physical attacks. By leveraging Multi-Agent Systems (MAS) and distributed learning, the framework aims to maintain operational normalcy even under duress.

The Evolving Threat Landscape

Traditional ICS were often isolated, centralized systems. Modern evolution has transformed them into "systems of systems" where Information Technology (IT) and Operational Technology (OT) are deeply interdependent. This convergence has significantly increased the attack surface, allowing Advanced Persistent Threats (APTs) to move laterally from business networks to physical controllers.

Hierarchical IT/OT architecture showing the propagation of cyber-physical attacks across levels.

In this environment, robustness—the ability to withstand known disturbances—is insufficient. Resilience is defined by the (T, D) metric, where T represents the time to recover and D represents the maximum loss of performance. A truly resilient system minimizes both variables in the face of unexpected, malicious disruptions.

A Multi-Agent Framework for Resilience

The proposed solution centers on an "AI Stack" integrated into a two-level MAS architecture. At the system level, individual subsystems act as intelligent agents; at the subsystem level, IT and OT units coordinate their actions through the AI Stack.

The multi-agent system architecture featuring integrated AI Stacks within each subsystem.

The AI Stack serves as the computational "brain" connecting cyber and physical agents. This architecture allows for distributed control, which is essential for massive, heterogeneous networks like smart grids where centralized management is no longer feasible.

Methodological Innovations: Co-Learning and Game Theory

A core contribution of this framework is Cyber-Physical Co-Learning. Unlike traditional security research that treats IT and OT in isolation, this approach mandates that IT learning be "impact-aware" (understanding how cyber-defense affects physical output) and OT learning be "cyber-aware" (recognizing potentially compromised sensor data).

To address intelligent adversaries, the framework employs Game-Theoretic Learning (e.g., Bayesian and Stackelberg games). By modeling attackers as rational players, the system can deploy "defensive deceptions" such as honeypots. Furthermore, Federated Learning allows subsystems to share intelligence without exposing sensitive raw data, while Meta-Learning enables new subsystems to rapidly adapt base models to their specific physical dynamics.

Case Study: Distributed Renewable Energy

The framework's efficacy was demonstrated through a wind energy case study using Markov Jump Linear Systems (MJLS) to simulate transitions between normal and attacked states. Key results include:

  • Enhanced Recovery: Co-learning enabled wind turbines to return to expected power output significantly faster than traditional controllers.
  • High Detection Accuracy: AI-augmented detection achieved precision rates exceeding 97% for known attack patterns.
  • Operational Risk Reduction: Reinforcement learning for human attention management reduced operational risk by up to 20% by filtering out deceptive "feint" attacks.

Conclusion and Future Directions

While the MAS framework offers a robust path toward resilient ICS, several challenges remain. These include optimizing sampling efficiency between high-frequency OT data and lower-frequency IT data, managing belief uncertainty in multi-agent interactions, and addressing the inherent heterogeneity of industrial hardware. Future work will focus on refining universal base models for diverse infrastructure components.

Summary updated

Abstract

Resilience describes a system's ability to function under disturbances and threats. Many critical infrastructures, including smart grids and transportation networks, are large-scale complex systems consisting of many interdependent subsystems. Decentralized architecture becomes a key resilience design paradigm for large-scale systems. In this book chapter, we present a multi-agent system (MAS) framework for distributed large-scale control systems and discuss the role of MAS learning in resiliency. This chapter introduces the creation of an artificial intelligence (AI) stack in the MAS to provide computational intelligence for subsystems to detect, respond, and recover. We discuss the application of learning methods at the cyber and physical layers of the system. The discussions focus on distributed learning algorithms for subsystems to respond to each other, and game-theoretic learning for them to respond to disturbances and adversarial behaviors. The book chapter presents a case study of distributed renewable energy systems to elaborate on the MAS architecture and its interface with the AI stack.
BibTeX
@misc{zhao2022multi,
  author = {Yuhan Zhao and Craig Rieger and Quanyan Zhu},
  title = {{Multi-Agent Learning for Resilient Distributed Control Systems}},
  year = {2022},
  eprint = {2208.05060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.05060},
}
Figure