Emergent Discrete Communication in Semantic Spaces

Mycal Tucker, Huao Li, Siddharth Agrawal, Dana Hughes, Katia P. Sycara, Michael Lewis, Julie Shah

In the field of multi-agent reinforcement learning (MARL), agents frequently need to communicate to coordinate complex tasks. Traditionally, researchers have forced agents to use one-hot vectors to represent discrete tokens (e.g., [1, 0, 0] for "apple", [0, 1, 0] for "banana").

However, one-hot vectors are mathematically orthogonal and equidistant. To an agent, the "distance" between "apple" and "banana" is identical to the distance between "apple" and "car." This lack of semantic structure makes communication fragile. If a message is corrupted by noise, the agent has no "nearby" concepts to fall back on. Furthermore, because these tokens lack inherent meaning, it is nearly impossible for humans to understand an agent's emergent language—or for an agent to understand a human—without extensive, specific training.

To address this, researchers have proposed moving beyond the one-hot standard by allowing neural agents to learn discrete tokens within a continuous semantic space. This "prototype-based" approach enables agents to communicate more robustly in noisy environments and, for the first time, allows for zero-shot understanding between agents and humans.

Prototype-Based Communication

Instead of simply outputting a one-hot index, the proposed agent's "communication head" utilizes a prototype-based architecture inspired by word embeddings in natural language processing (like Word2Vec or GloVe):

  1. Logit Generation: The agent produces a probability distribution over possible tokens.
  2. Discretization: Using the Gumbel-Softmax trick, it selects a discrete "prototype" index.
  3. Embedding: This index is multiplied by a learnable matrix $T$, mapping the discrete choice into a $c$-dimensional continuous vector (the "prototype").

Because these prototypes are learnable, agents can arrange them in space so that similar meanings are physically close to each other. A decision-theoretic analysis provided by the authors proves that one-hot vectors are only optimal under very narrow conditions (like zero noise and perfectly uniform task distributions). In more realistic scenarios, prototype-based communication is theoretically superior.

Experimental Environments

The paper evaluates this method across three distinct settings, including custom 2D navigation tasks known as Particle-World Environments. In these tasks, a "speaker" agent must guide a "listener" agent to a target landmark.

Particle-World Environments

Beyond the Particle-World navigation tasks, the researchers also utilized CIFAR-10 for a zero-shot reference game (where agents communicated about images of animals and vehicles) and Human-Generated Data via Amazon Mechanical Turk (AMT) captions to test if agents could understand natural language "out of the box" by aligning their prototypes with BERT embeddings.

Key Insights and Results

The shift to a continuous semantic space yields several critical advantages:

Semantic Clustering: Agents naturally learn to cluster tokens. In a navigation task, tokens for "top-left" and "top-right" landmarks end up closer to each other in the semantic space than to the token for "bottom."

Semantic Clustering in 3D Space

Graceful Degradation: In noisy channels, prototype agents significantly outperform one-hot agents. If a "top-left" signal is slightly distorted, the listener is still likely to move toward the top, whereas a one-hot agent would be completely confused. In the "Triangle" environment with high noise, prototype agents achieved a median reward of -124, significantly better than the -201 achieved by one-hot agents.

Zero-Shot "Aha!" Moments: By initializing prototypes with pre-trained word embeddings (like BERT), agents can understand human phrases they have never seen before. In the CIFAR-10 reference game, prototype agents using BERT embeddings achieved 58-66% accuracy on images from classes they were never trained on, whereas one-hot agents performed at random chance (50%).

Human Interpretability: Because the agent's language has a spatial structure, humans can interpret unlabeled agent tokens by looking at their neighbors in the embedding space. In a user study, humans correctly identified an agent's intended target image 70% of the time just by looking at where the agent's emergent token sat in a 2D semantic map.

Limitations and Future Work

While highly effective, this method adds a layer of complexity: agents must learn the meaning of tokens and the coordinates of those tokens simultaneously, which can increase training time. Additionally, natural language is highly compositional. A major future direction is applying this prototype approach to multi-token sequences to see if agents can develop a "grammar" that humans can intuitively navigate. Finally, researchers must remain cautious, as using pre-trained embeddings can inadvertently bake existing human biases into agent behavior.

Summary updated

Abstract

Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.
BibTeX
@inproceedings{tucker2021emergent,
  author = {Mycal Tucker and Huao Li and Siddharth Agrawal and Dana Hughes and Katia P. Sycara and Michael Lewis and Julie Shah},
  title = {{Emergent Discrete Communication in Semantic Spaces}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=hsqZ5v8PFyQ},
}
Figure