Learning to Communicate and Solve Visual Blocks-World Tasks.

Qi Zhang, Richard L. Lewis, Satinder P. Singh, Edmund H. DurfeeAAAI 2019 · OpenReview

Abstract

We study emergent communication between speaker and listener recurrent neural-network agents that are tasked to cooperatively construct a blocks-world target image sampled from a generative grammar of blocks configurations. The speaker receives the target image and learns to emit a sequence of discrete symbols from a fixed vocabulary. The listener learns to construct a blocks-world image by choosing block placement actions as a function of the speaker’s full utterance and the image of the ongoing construction. Our contributions are (a) the introduction of a task domain for studying emergent communication that is both challenging and affords useful analyses of the emergent protocols; (b) an empirical comparison of the interpolation and extrapolation performance of training via supervised, (contextual) Bandit, and reinforcement learning; and (c) evidence for the emergence of interesting linguistic properties in the RL agent protocol that are distinct from the other two.

Introduction

The field of emergent communication explores how artificial agents can develop their own protocols to solve tasks without pre-defined human language. While previous research often focused on simple referential games—where one agent describes an image for another to identify—this work tackles the more complex challenge of sequential construction. In this setup, a "Speaker" observes a target block structure and must communicate a construction plan to a "Listener" who cannot see the target. This requires the agents to develop a language capable of describing complex spatial plans and generalizing to novel, larger structures.

Methodology

The researchers utilize a dual-agent architecture consisting of a Speaker and a Listener. Both agents employ Convolutional Neural Networks (CNNs) for visual processing and Long Short-Term Memory (LSTM) networks for handling symbol sequences. To bridge the gap between continuous neural gradients and discrete communication symbols, the Gumbel-Softmax estimator is used during training.

The Speaker-Listener architecture featuring CNNs for vision and LSTMs for sequential communication.

Three training regimes were compared to evaluate their impact on the emergent language:

  1. Supervised Learning (SL): Agents are trained to mimic a fixed, canonical construction order.
  2. Bandit Training: Agents receive immediate feedback (+1 or -1) after every block placement.
  3. Reinforcement Learning (RL): Agents receive a sparse reward only upon the completion of the entire structure, creating a significant credit-assignment challenge.

The Blocks-World Environment

The task environment is a $10 \times 10$ grid where agents place blocks of three sizes: small ($1\times1$), medium ($2\times2$), and large ($3\times3$). A probabilistic context-free grammar (PCFG) generates a vast variety of target configurations, ensuring the agents must learn generalizable principles rather than memorizing specific patterns.

Examples of diverse block configurations generated by the probabilistic context-free grammar.

Emergent Linguistic Properties

One of the most significant findings is that the "hardest" training path—Reinforcement Learning—yielded the most sophisticated linguistic structures. Despite the difficulty of delayed rewards, RL agents developed protocols with several human-like properties:

  • Functional Phrases: RL agents developed specific symbol sequences (N-grams) that functioned as cohesive units or "words," appearing with much higher frequency than chance.
  • Zipf’s Law: The frequency distribution of these emergent phrases followed a power-law distribution, a mathematical signature found in all natural human languages.
  • Compositionality: RL protocols showed higher compositionality, meaning the total message was more clearly built from the meanings of its parts. This was demonstrated by the fact that RL agents were more robust when their utterances were truncated compared to SL or Bandit agents.

Performance and Generalization

In terms of task performance, both Bandit and RL training significantly outperformed Supervised Learning when extrapolating to larger structures (up to 40 blocks) than those seen during training (up to 25 blocks). This suggests that learning through trial-and-error allows agents to discover more flexible and robust communication strategies than simple imitation of a fixed order.

Furthermore, the agents demonstrated an internal understanding of the environment's "physics." Even when they failed to recreate the target perfectly, 80% of their incorrect actions still followed the underlying rules of the blocks-world grammar, such as avoiding floating blocks.

Conclusion and Future Work

While RL agents developed the most linguistically interesting protocols, they faced greater difficulty in overall task completion on the most complex structures compared to Bandit agents, likely due to the sparse reward signal. Future research may explore scaling these systems to allow for multi-turn dialogue, applying advanced unsupervised linguistic analysis to further decode the emergent protocols, and investigating the feasibility of human-AI communication using these evolved languages.

Summary updated

Abstract

We study emergent communication between speaker and listener recurrent neural-network agents that are tasked to cooperatively construct a blocks-world target image sampled from a generative grammar of blocks configurations. The speaker receives the target image and learns to emit a sequence of discrete symbols from a fixed vocabulary. The listener learns to construct a blocks-world image by choosing block placement actions as a function of the speaker’s full utterance and the image of the ongoing construction. Our contributions are (a) the introduction of a task domain for studying emergent communication that is both challenging and affords useful analyses of the emergent protocols; (b) an empirical comparison of the interpolation and extrapolation performance of training via supervised, (contextual) Bandit, and reinforcement learning; and (c) evidence for the emergence of interesting linguistic properties in the RL agent protocol that are distinct from the other two.
BibTeX
@inproceedings{zhang2019learning,
  author = {Qi Zhang and Richard L. Lewis and Satinder P. Singh and Edmund H. Durfee},
  title = {{Learning to Communicate and Solve Visual Blocks-World Tasks.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=hAqBKEb-eV},
}
Figure