Reinforcement Learning Based Multi-Agent Resilient Control: From Deep Neural Networks to an Adaptive Law

Jian Hou, Fangyuan Wang, Lili Wang, Zhiyong ChenAAAI 2021 · DOI

Abstract

Multi-agent systems, such as autonomous drone fleets or smart power grids, require robust mechanisms to reach consensus even when some nodes are faulty or malicious. This paper proposes two reinforcement learning (RL) based approaches—Distributed-DDPG (D-DDPG) and a lightweight Q-consensus algorithm—to address the limitations of traditional resilient control methods. By treating neighbor weights as learnable parameters, agents can identify and isolate "bad actors" through trial and error. Experimental results demonstrate that these methods achieve high success rates in sparse networks where traditional algorithms typically fail.

Introduction

In Multi-Agent Resilient Consensus (MARC), the objective is for a group of "normal" agents to agree on a common state (e.g., velocity, heading, or sensor value) despite the presence of "faulty" agents that inject false or disruptive data. Traditional solutions, such as the Weighted-Mean Subsequence-Reduced (W-MSR) algorithm, often require highly dense network topologies to mathematically filter out bad data. Specifically, to tolerate $F$ faulty neighbors, a node typically needs a large number of total connections, which is often impractical in real-world deployments. Furthermore, these methods often assume the number of faults is known a priori.

This research explores whether agents can learn to distinguish between reliable teammates and malicious opponents by observing how a neighbor's data impacts their progress toward the collective goal.

Methodology

The authors propose a two-stage evolution of RL-based resilient control:

1. Distributed-DDPG (D-DDPG)

The first approach utilizes a Deep Deterministic Policy Gradient (DDPG) framework. Each agent operates its own actor-critic neural network where the "action" is the weight assigned to each neighbor. A reward function is defined based on the relative distance between agents; neighbors whose data helps an agent move closer to the group consensus yield higher rewards, while those causing deviation are penalized with lower rewards.

2. Q-consensus

Recognizing the high computational cost of deep neural networks for real-time control, the authors distilled the RL logic into a simpler "adaptive law" called Q-consensus. This algorithm uses a credibility function, inspired by Q-learning, to recursively update neighbor weights. This lightweight mathematical rule replaces the heavy neural network while retaining the adaptive "trial and error" capability.

State convergence under Q-consensus with various faulty node configurations.

Experimental Evaluation

The algorithms were evaluated through numerical simulations and hardware benchmarks:

  • Numerical Simulations: A 10-node directed graph was used to test resilience against various fault types, including persistent random noise, constant values, and malicious "learnable" agents.
  • Hardware Benchmark: A real-world testbed with six Corey Mecanum wheel vehicles verified velocity synchronization under faulty conditions.

Key Results

Both D-DDPG and Q-consensus achieved near 100% success rates across most fault configurations. Notably, Q-consensus significantly outperformed D-DDPG in convergence speed, reaching consensus in far fewer steps by avoiding the lengthy training phase required for deep neural networks.

Success rate comparison between D-DDPG and Q-consensus across different faulty node types.

Discussion and Conclusion

The core innovation of this work is the use of reward as a proxy for trust, allowing agents to break the "topology barrier" inherent in traditional methods. By learning behavior-based weights, the system can operate effectively in sparse networks. The transition from D-DDPG to Q-consensus highlights a valuable path for AI in control: using deep learning for discovery and then simplifying the resulting intuition into efficient adaptive laws for deployment.

Future work will address limitations such as fixed network topologies, communication delays, and the potential for intrinsic rewards to further accelerate learning in complex environments.

Summary updated

BibTeX
@inproceedings{hou2021reinforcement,
  author = {Jian Hou and Fangyuan Wang and Lili Wang and Zhiyong Chen},
  title = {{Reinforcement Learning Based Multi-Agent Resilient Control: From Deep Neural Networks to an Adaptive Law}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i9.16945},
}

Loading related content...

Figure