Solving the Diffusion of Responsibility Problem in Multiagent Reinforcement Learning with a Policy Resonance Approach

Qingxu Fu, Tenghai Qiu, Jianqiang Yi, Zhiqiang Pu, Xiaolin Ai, Wanmai Yuan

Introduction

The "bystander effect" or "diffusion of responsibility" (DR) is a well-known social psychological phenomenon where individuals are less likely to offer help when others are present. This research reveals a strikingly similar failure mode in Multiagent Reinforcement Learning (MARL). When large groups of agents ($N > 30$) learn to cooperate, they frequently struggle with "minority responsibility"—tasks that only a few agents need to perform for the team to succeed. For instance, in a scenario requiring one agent to guard a gate while others attack, standard algorithms often fail because every agent "expects" another to fulfill the role.

The Root of the Problem

The authors argue that DR in MARL is not primarily a credit assignment or reward calculation issue. Instead, it is rooted in the exploration-exploitation dilemma. In standard MARL, agents explore the environment independently and simultaneously. Consequently, the "noise" from one agent's exploration is perceived by others as random environmental instability. As the team size grows, this aggregate noise becomes overwhelming. Agents, unable to distinguish between environmental randomness and teammate exploration, settle into safe "mainstream" behaviors, ignoring critical but risky minority roles.

Proposed Solution: Policy Resonance (PR)

To address this, the authors propose Policy Resonance (PR), a training-time plugin compatible with state-of-the-art algorithms like PPO or QMIX. PR forces agents to synchronize their decision-making states during Centralized Training (CT). The system alternates between two states based on a resonance probability ($\eta$):

  1. Resonated State: All agents act "greedily" according to their current best policy, pausing exploration to focus on exploitation.
  2. Non-Resonated State: Agents act autonomously and explore independently, following standard procedures.
The Policy Resonance mechanism synchronizes agent states at either the episode or step level during training.

By periodically forcing the team to act in unison without the "fidgeting" of individual exploration, the mutual influence of exploration is resolved. This clarity allows learning algorithms to identify the benefits of specialized minority behaviors. Crucially, resonance is only applied during training; during execution, agents remain fully decentralized, maintaining the Centralized Training, Decentralized Execution (CTDE) paradigm.

Experimental Evaluation

The approach was evaluated in two primary environments:

  • Fast Multiagent Evaluation (FME): A diagnostic suite featuring tasks like Lv-C (Responsibility Assignment). This level simulates a "fragile bridge" that collapses if too many agents attempt to cross simultaneously, specifically testing for DR.
  • Asymmetric Decentralized Collective Assault (ADCA): A high-fidelity swarm combat environment where 50 RL agents face an outnumbering force of scripted AI.

Key Results

In the FME Lv-C task, standard PPO-MA failed completely as the team size reached 7 agents. However, with Policy Resonance, teams maintained optimal performance even with 40 agents. In the ADCA swarm combat task ($N=50$), adding PR to the Conc4Hist algorithm increased the win rate from approximately 50% to over 80%.

Performance comparison across different agent counts (N=15 to 40) showing the significant reward boost when the PR stage begins.

The authors also introduced PR-Fast, a variant that freezes the "body" of the neural network during the resonance phase, achieving similar performance gains with reduced computational overhead.

Conclusion

Policy Resonance provides a simple yet effective solution to the diffusion of responsibility problem in large-scale MARL. By recognizing that independent exploration acts as noise for teammates, PR enables agents to "agree" on specialized roles through synchronized training phases. This approach paves the way for more effective coordination in complex, many-agent systems without sacrificing decentralized execution.

Summary updated

Abstract

State-of-the-art (SOTA) multiagent reinforcement algorithms distinguish themselves in many ways from their single-agent equivalences. However, most of them still totally inherit the single-agent exploration-exploitation strategy. Naively inheriting this strategy from single-agent algorithms causes potential collaboration failures, in which the agents blindly follow mainstream behaviors and reject taking minority responsibility. We name this problem the responsibility diffusion (RD) as it shares similarities with the same-name social psychology effect. In this work, we start by theoretically analyzing the cause of this RD problem, which can be traced back to the exploration-exploitation dilemma of multiagent systems (especially large-scale multiagent systems). We address this RD problem by proposing a policy resonance (PR) approach which modifies the collaborative exploration strategy of agents by refactoring the joint agent policy while keeping individual policies approximately invariant. Next, we show that SOTA algorithms can equip this approach to promote the collaborative performance of agents in complex cooperative tasks. Experiments are performed in multiple test benchmark tasks to illustrate the effectiveness of this approach.
BibTeX
@article{fu2025solving,
  author = {Qingxu Fu and Tenghai Qiu and Jianqiang Yi and Zhiqiang Pu and Xiaolin Ai and Wanmai Yuan},
  title = {{Solving the Diffusion of Responsibility Problem in Multiagent Reinforcement Learning with a Policy Resonance Approach}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2024.3423417},
}
Figure