Decentralized Multi-Agent Reinforcement Learning via Consensus and Innovations

This direction is merged into another direction: partial observability (partial observability in multi-agent systems). Its last version stays readable and citable here.

Version 1 of 116 papers17 conceptsfrozen 2026-10-04

In traditional reinforcement learning, we typically imagine a single agent interacting with an environment. Even in multi-agent systems, many algorithms rely on a "centralized" perspective where a single controller observes every agent’s state and action. However, in large-scale systems like robot swarms or smart power grids, this central bottleneck is often impractical or impossible.

The research direction of decentralized multi-agent reinforcement learning (MARL) seeks to enable a network of agents to collectively learn an optimal global policy using only local observations and sparse, neighbor-to-neighbor communication.

The Consensus plus Innovations Framework

The foundational challenge in decentralized learning is how to ensure that agents, each seeing only a fraction of the world, can agree on a global strategy. José M. F. Moura introduced a rigorous framework to solve this known as $\mathcal{QD}$-Learning.

The intuition behind $\mathcal{QD}$-learning lies in two simultaneous processes:

  1. Consensus: Agents communicate with their immediate neighbors to average their estimates of the "value" of certain actions. This prevents any single agent from diverging too far from the group's collective knowledge.
  2. Innovations: While reaching a consensus, each agent also incorporates "innovations"—new information derived from its own local interactions with the environment.

By balancing these two forces, the network can solve a Markov Decision Process (MDP) collectively. The "Consensus" step ensures the agents act as a unified team, while the "Innovations" step ensures the team actually learns from the environment.

Stabilizing the Moving Target

A notorious problem in multi-agent systems is that the environment becomes "non-stationary." Because every agent is learning and changing its behavior simultaneously, the optimal strategy for Agent A is constantly shifting as Agent B updates its policy.

To address this, Bora Yongacoglu, Gürdal Arslan, and Serdar Yuksel proposed a two-timescale approach. In this setup:

  • On a fast timescale, agents evaluate their current strategies to see how well they are performing.
  • On a slow timescale, agents update their baseline behaviors.

This separation of concerns allows the system to stabilize. It leads to a concept called Subjective Q-equilibrium, where agents reach a stable state that accounts for their limited individual perspectives while still achieving a globally optimal outcome.

Optimizing Complex Goals with Shadow Rewards

Most RL algorithms focus on maximizing a simple sum of rewards. However, real-world teams often have complex requirements, such as safety constraints, risk-aversion, or a desire for maximum exploration.

Amrit Bedi, Alec Koppel, and Mengdi Wang extended decentralized learning to these "general utilities" through a mechanism called Shadow Reward Actor-Critic (DSAC). Instead of just sharing raw rewards, agents maintain "shadow" variables that track how well the team is meeting complex global objectives. This allows a decentralized swarm to optimize for non-linear goals that would normally require a central supervisor.

Efficiency and the Frontier of Partial Observability

As these systems scale to thousands of agents, communication becomes a bottleneck. Ziyi Chen and colleagues have developed communication-efficient algorithms that use mini-batch sampling to reduce the frequency of messages between agents without sacrificing the speed of learning. Similarly, Chao Qu and Shie Mannor introduced Value Propagation, which uses deep neural networks to approximate these value functions in a fully decentralized way.

Despite these advances, a significant challenge remains. Most current proofs for global optimality assume that while agents are decentralized, they can still observe a "global state" (e.g., every robot knows the position of every other robot). A recurring theme across the work of Yongacoglu and Bedi is the push toward purely local observability. In this more difficult setting, agents must coordinate based on incomplete and potentially conflicting local perceptions—a frontier that remains the primary hurdle for truly autonomous decentralized systems.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decentralized_multi_agent_reinforcement__061dcf59,
  title = {Decentralized Multi-Agent Reinforcement Learning via Consensus and Innovations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decentralized-multi-agent-reinforcement-learning-via-consensus-and-innovations/rdr_061dcf59c6194cc4b2a15d329f8f4632},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decentralized_multi_agent_reinforcement__061dcf59_v1,
  title = {Decentralized Multi-Agent Reinforcement Learning via Consensus and Innovations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decentralized-multi-agent-reinforcement-learning-via-consensus-and-innovations/rdr_061dcf59c6194cc4b2a15d329f8f4632/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}