Learning to Learn to Communicate

Abhinav Gupta, Ryan Lowe, Jakob Nicolaus Foerster, Douwe Kiela, Joelle PineauOpenReview 2019 · OpenReview

TL;DR

This paper introduces the Learning to Learn to Communicate (L2C) framework, a meta-learning approach to train agents that can quickly adapt to and learn new human languages. By pre-training a meta-learner in simulation with diverse agent populations, L2C aims to create agents that require significantly fewer human interactions to acquire language skills, mimicking how human babies learn.

What Problem Does This Paper Address?

The core challenge this paper tackles is the data inefficiency of current machine learning methods when it comes to teaching artificial agents to understand and use human language flexibly in real-world tasks. While humans learn language effectively through interaction and a "human-in-the-loop" process, existing AI approaches struggle to achieve this level of sample efficiency. This makes it difficult and expensive to train agents for tasks requiring nuanced communication, especially when aiming for human-like language acquisition. The paper seeks to bridge this gap by developing a "language learning prior" that enables agents to learn new languages with minimal human guidance.

Methods

The proposed solution is the Learning to Learn to Communicate (L2C) framework, which operates in three phases:

  1. Train Agent Populations: Agents are trained in simulated environments to solve tasks using emergent communication. This phase involves creating diverse populations of agents, each developing its own unique communication protocol.
  2. Train Meta-Learner on Agent Populations: A meta-learning agent is trained to quickly adapt to the communication protocols of these diverse agent populations. The goal is for the meta-learner to achieve high performance with only a few updates on each new population.
  3. Test Meta-Learner: The meta-learner is then evaluated on its ability to learn new languages, which can be unseen artificial languages or, crucially, human languages.

The key innovation is the meta-learning phase (Phase 2), where the agent learns how to learn languages rather than just learning a specific language. This meta-training in simulation leverages abundant compute resources to build a robust "language learning prior."

Datasets

The paper describes experiments using a simulated speaker-listener game, similar to the "Task & Talk" game. This game involves:

  • Objects with properties: Defined by shape and color (e.g., p properties, t types per property).
  • Programmatically generated compositional language: To approximate human language, concepts (like "blue" or "triangle") are assigned unique symbols, allowing for a structured and compositional language. This avoids the need for human-annotated data during the initial meta-training phases.
  • Two game sizes: A smaller game (p=3, t=5) and a larger game (p=6, t=10) were used to test generalization.

Key Insights and Ideas

  • Meta-learning as a "language learning prior": The core idea is that by training an agent to rapidly adapt to many different communication systems, it develops a general ability to learn new languages, rather than just mastering one.
  • Simulation for efficient meta-training: Leveraging simulation allows for massive pre-training of the meta-learner on diverse communication protocols without the cost of human interaction.
  • Generalization to "human-like" language: The experiments suggest that agents trained with L2C generalize better to a perfectly compositional, human-like language than to other emergent languages, highlighting the importance of structure.
  • The value of diverse communication partners: Exposing the meta-learner to a wide variety of communication protocols during training is crucial for its ability to adapt to new ones.

Key Results

  • Agents trained with the L2C framework (using meta-learning algorithms like FOMAML and Reptile) learned a simple compositional language in significantly fewer iterations than randomly initialized agents.
  • Figure 2 shows that the meta-learner required substantially fewer updates (iterations) to reach 95% reconstruction performance on held-out test populations compared to a random baseline. For example, in the "compositional bot" (CB) scenario, the meta-learner achieved this performance much earlier in training.
  • The paper notes that agents trained with L2C generalized better to the "human" language (which was perfectly compositional) than to other emergent languages.

Limitations and Future Work

  • Meta-training via RL: The current meta-training is done via supervised learning. The authors suggest training the meta-agent via Reinforcement Learning (RL) for more complex scenarios.
  • Joint Speaker-Listener: The current setup focuses on a listener meta-agent. Future work will explore training a joint speaker-listener meta-agent that can both produce and understand language.
  • Scaling to complex tasks: The current experiments are in a simplified speaker-listener game. Scaling L2C to more complex, grounded language learning tasks (like navigation) is a key future direction.
  • Understanding generalization properties: A crucial open question is understanding what properties of the training populations are essential for the meta-learner to generalize to human language.
  • Incorporating human demonstrations: The authors are excited about combining meta-imitation learning with RL and incorporating existing human language datasets to further improve sample efficiency.
  • Phase 1 challenges: The paper acknowledges difficulties in Phase 1 (training agent populations), particularly the non-stationarity and high variance in multi-agent RL for emergent communication. They suggest solutions like differentiable environments or seeding with human demonstrations.

Summary updated

Abstract

How can we teach artificial agents to use human language flexibly to solve problems in a real-world environment? We have one example in nature of agents being able to solve this problem: human babies eventually learn to use human language to solve problems, and they are taught with an adult human-in-the-loop. Unfortunately, current machine learning methods (e.g. from deep reinforcement learning) are too data inefficient to learn language in this way. An outstanding goal is finding an algorithm with a suitable ‘language learning prior’ that allows it to learn human language, while minimizing the number of required human interactions. In this paper, we propose to learn such a prior in simulation, leveraging the increasing amount of available compute for machine learning experiments. We call our approach Learning to Learn to Communicate (L2C). Specifically, in L2C we train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. Once the meta-learning agent is able to quickly adapt to each population of agents, it can be deployed in new populations unseen during training, including populations of humans. To show the promise of the L2C framework, we conduct some preliminary experiments in a Lewis signaling game, where we show that agents trained with L2C are able to learn a simple form of human language (represented by a hand-coded compositional language) in fewer iterations than randomly initialized agents.
BibTeX
@misc{gupta2019learning,
  author = {Abhinav Gupta and Ryan Lowe and Jakob Nicolaus Foerster and Douwe Kiela and Joelle Pineau},
  title = {{Learning to Learn to Communicate}},
  year = {2019},
  url = {https://openreview.net/forum?id=Skgi2oSIdH},
  howpublished = {OpenReview},
}
Figure