Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition

Dylan Cope, Peter McBurneyIJCAI 2024

Introduction

A significant challenge in multi-agent reinforcement learning is the "private" nature of emergent communication. When agents evolve their own protocols to solve tasks, these languages are often unintelligible to new agents attempting to join the group. This phenomenon, known as Zero-Shot Coordination (ZSC), typically assumes no prior knowledge of the target group. However, in many practical scenarios, we possess logs of past interactions. This paper, presented at IJCAI-24, addresses the Cooperative Language Acquisition Problem (CLAP): how a joiner agent can leverage such datasets to learn the syntax and semantics of a pre-existing communication system.

Methodology: ECTL vs. Behavior Cloning

The authors propose Emergent Communication pretraining and Translation Learning (ECTL) to bridge the gap between ad hoc teamwork and zero-shot coordination. ECTL operates in two distinct phases:

  1. Pretraining: The joiner agent undergoes self-play in the environment to develop its own internal emergent language and master the task's mechanical requirements (e.g., navigation or driving).
  2. Translation: The agent then uses the expert dataset to learn a translation layer. This layer maps the expert community's messages to the agent's internal "concepts" and vice versa, rather than learning the task from scratch.

In contrast, the Behavior Cloning (BC) baseline treats the problem as a standard supervised learning task. It attempts to map observations directly to expert messages (signaling) and received messages to expert actions (listening).

The Behavior Cloning (BC) Backward architecture used as a baseline for comparison.

Experimental Setup

The effectiveness of ECTL was evaluated across three diverse setups:

  • Goal Communications Gridworld: A 5x5 grid where agents must coordinate to reach specific targets. Communication is essential as agents only know their partner's goal.
  • Goal Communications Driving Game: A continuous navigation task featuring a high-penalty "pit" in the center, designed to test robustness against biased expert data.
  • Human-in-the-loop Data: A dataset derived from human-controlled agents, where messages represent direct instructions.
Overview of the joint agent architecture and the two primary experimental environments: gridworld and driving.

Key Insights and Results

The core advantage of ECTL lies in disentangling skills from protocol. By mastering the environment's "physics" during self-play, the agent only needs the expert data to learn "how to talk," not "how to play."

Robustness to Biased Data

Behavior Cloning is often brittle; if expert data is narrow (e.g., experts never enter the "pit"), a BC agent cannot recover from errors. ECTL, having explored the full state space during pretraining, remains robust. In the Gridworld task, restricting training data to specific columns caused BC's performance to plummet, while ECTL maintained near-perfect accuracy.

Data Efficiency and Semantic Capture

ECTL proves significantly more data-efficient than BC. Because the agent already understands the world's underlying logic, it requires fewer examples to decipher the community's "dictionary." Furthermore, human-in-the-loop experiments showed that ECTL successfully captured the semantics of instructions, allowing human listeners to reach goals faster when using ECTL-translated messages compared to BC or random baselines.

Conclusion

ECTL represents a shift from pure imitation to translation-based learning in multi-agent systems. By separating task mastery from communicative alignment, it provides a robust framework for agents to join existing communities and adapt to their unique languages, even when faced with limited or biased interaction data.

Summary updated

BibTeX
@inproceedings{cope2024learning,
  author = {Dylan Cope and Peter McBurney},
  title = {{Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure