Abstract
Communication is essential for coordination among humans and animals. Therefore, with the introduction of intelligent agents into the world, agent-to-agent and agent-to-human communication becomes necessary. In this paper, we first study learning in matrix-based signaling games to empirically show that decentralized methods can converge to a suboptimal policy. We then propose a modification to the messaging policy, in which the sender deterministically chooses the best message that helps the receiver to infer the sender's observation. Using this modification, we see, empirically, that the agents converge to the optimal policy in nearly all the runs. We then apply this method to a partially observable gridworld environment which requires cooperation between two agents and show that, with appropriate approximation methods, the proposed sender modification can enhance existing decentralized training methods for more complex domains as well.
Introduction
In cooperative multi-agent reinforcement learning (MARL), agents often struggle to develop effective communication protocols when messages don't directly impact rewards—a setting known as "cheap talk." Standard RL algorithms, such as Independent Q-Learning (IQL), frequently get stuck in sub-optimal equilibria. This paper identifies a "vicious cycle": if a receiver agent learns to take a "safe" but sub-optimal action when uncertain, the sender agent loses the incentive to send distinct, informative messages. Consequently, the receiver never learns the optimal, state-specific actions. To address this, we introduce an inference-based messaging approach where the sender simulates the receiver's Bayesian inference to choose the most informative message, breaking sub-optimal learning cycles.
Methods
The proposed Inference-Based Messaging modifies the sender's policy to actively help the receiver infer its private state.
Tabular Case (Exact Inference)
The sender calculates the posterior probability of its current state $s$ given each possible message $m$, denoted as $p(s|m)$. It then deterministically sends the message that maximizes this posterior. Intuitively, the sender asks itself, "If the receiver uses Bayes' rule on my message, which message makes my true state the most obvious?"
Approximation Case (Gridworlds)
In complex, partially observable environments, exact Bayesian inference is intractable. We use an LSTM to summarize the agent's history into a hidden state representation. To approximate the marginal probability of a message $p(m)$, we use an exponentially weighted moving average of the empirical messaging probabilities over rollouts.
Training
Underlying policies are trained using standard RL algorithms: Q-Learning for tabular games and the off-policy actor-critic algorithm IMPALA for gridworld tasks.
Key Insights: The "Safe Action" Trap
Empirical analysis reveals that standard MARL algorithms fail because receivers default to actions with low penalties for error. This lack of consequence removes the gradient signal needed for the sender to differentiate its messages.
By forcing the sender to maximize the posterior probability of its state, the algorithm provides an intrinsic motivation to fully utilize the communication channel, regardless of immediate reward. This breaks the cycle and allows the receiver to learn optimal responses to distinct messages.
Results
The method was evaluated on synthetic cooperative tasks, including the Climbing Game, random signaling games, and a Treasure Hunt Gridworld.
In signaling games, the proposed Info-Q method converged to the optimal policy in nearly 100% of runs, whereas baselines like IQL, Lenience, and Hysteretic-Q frequently failed. In the Treasure Hunt Gridworld, Info-Q achieved a mean final reward of 14.29, significantly outperforming the IMPALA baseline (11.55). Combining Info-Q with a "positive listening bias" further improved performance to a mean reward of 15.48.
Limitations and Future Work
Approximating posterior probabilities in complex environments remains challenging. The current moving average method for $p(m)$ suffers from high variance due to small batch sizes. Future work will focus on developing more sophisticated inference approximation techniques for high-dimensional, continuous state spaces to unlock stronger performance in complex multi-agent domains.
Summary updated