Emergent Discrete Communication in Semantic Spaces

Mycal Tucker, Huao Li, Siddharth Agrawal, Dana Hughes, Katia P. Sycara, Michael Lewis, Julie Shah

Abstract

Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.

Introduction

In the field of emergent communication, researchers train neural agents to solve tasks by "talking" to one another. Traditionally, this communication is discretized using one-hot vectors. While effective for simple tasks, one-hot vectors have a major flaw: they are all orthogonal and equidistant. In a one-hot lexicon, the token for "dog" is just as different from "cat" as it is from "car."

This lack of internal structure prevents agents from learning semantic relationships. It makes communication brittle—a single bit of noise can turn a meaningful message into something completely unrecognizable. Furthermore, it makes "zero-shot" understanding (interpreting a message you've never seen before) impossible, as there is no basis for interpolation or similarity.

Methodology: Prototype-Based Communication

The authors propose a prototype-based communication architecture. Instead of just outputting a one-hot index, the agent's communication head learns a "lexicon" matrix $T$.

  1. Architecture: The agent generates logits from an observation, which are converted into a one-hot vector $d$ using the Gumbel-Softmax trick (allowing for differentiable discrete selection).
  2. The "Prototype": This vector $d$ is then multiplied by the learnable matrix $T$. The resulting token $t$ is a dense vector in a continuous space.
  3. Decision-Theoretic Analysis: A mathematical derivation shows that one-hot vectors are only optimal under very restrictive assumptions (like an identity reward matrix and uniform priors). In the real world, where some mistakes are "better" than others (e.g., confusing a dog for a cat vs. a car), semantic embeddings are theoretically superior.
Visualization of the mapping from a 3D semantic embedding space to discrete tokens in the 9 Points environment.

Experimental Setup

The researchers evaluated their approach across several environments, primarily focusing on Particle-World Environments: three custom scenarios (Triangle, 9 Points, and Uniform) where a "speaker" agent must guide a "listener" agent to specific landmarks. These were used to test noise robustness and the ability to handle non-uniform target distributions.

Particle-World environments used for evaluation: (a) Triangle, (b) 9 Points, and (c) Uniform.

Beyond these synthetic tasks, the agents were tested on CIFAR-10 in a reference game setting and against Human Annotations (1,257 valid captions) to evaluate zero-shot understanding of natural human descriptions.

Key Insights and Results

Robustness and Clustering

In noisy environments, prototype-based agents naturally learn to cluster similar meanings together in the embedding space. If noise nudges a "dog" token, it likely still lands near other "animal" tokens, allowing the listener to make a "reasonable" mistake rather than failing entirely. In the "Triangle" environment with high noise ($\sigma=0.9$), prototype agents achieved a median reward of -124, significantly outperforming one-hot agents at -201.

Zero-Shot Understanding

Because the tokens exist in a continuous space, agents can respond to novel vectors. In the "Uniform" environment, agents showed they could interpret a vector halfway between two known prototypes as a request to go to a location halfway between the corresponding landmarks. When presented with novel human descriptions (AMT captions) for CIFAR-10 images, prototype agents achieved 56% accuracy, whereas one-hot agents were limited to random chance (~50%).

Human-Interpretable Emergence

Even without explicit language training, the "lexicons" learned by agents often mirrored human semantic categories (e.g., grouping all vehicle types together). In a study where humans guessed which image an agent was "talking" about based on a 2D map of the agent's tokens, humans were 70% accurate, outperforming even a BERT-based baseline (66%).

Conclusion

This work demonstrates that moving beyond one-hot vectors to semantic embedding spaces allows emergent communication to be more robust, flexible, and human-interpretable. While training instability in multi-agent reinforcement learning remains a challenge, the ability to exploit environmental priors and enable zero-shot understanding marks a significant step forward for agent-to-agent and agent-to-human interaction.

Summary updated

Abstract

Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.
BibTeX
@inproceedings{tucker2021emergent,
  author = {Mycal Tucker and Huao Li and Siddharth Agrawal and Dana Hughes and Katia P. Sycara and Michael Lewis and Julie Shah},
  title = {{Emergent Discrete Communication in Semantic Spaces}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=hsqZ5v8PFyQ},
}
Figure