Linking Emergent and Natural Languages via Corpus Transfer

Shunyu Yao, Mo Yu, Yang Zhang, Karthik R. Narasimhan, Joshua B. Tenenbaum, Chuang Gan

TL;DR

This paper bridges the gap between AI-generated "emergent languages" and human natural languages by introducing corpus transfer: pre-training models on AI-speak to improve performance on human-speak tasks. The authors demonstrate that emergent languages possess structural properties that significantly benefit low-resource NLP tasks. Furthermore, they propose a new translation-based metric that accurately predicts an emergent language's "human-likeness" and utility, challenging previous evaluation standards.

In the field of Emergent Communication (EC), researchers train AI agents to develop their own protocols (languages) to solve collaborative tasks. A classic example is a "referential game" where one agent must describe an image so another can pick it out of a lineup.

Historically, these emergent languages have been studied in a vacuum. Researchers relied on internal metrics like "game success" or "topographic similarity" (how well the language's structure maps to object attributes) to evaluate them. However, it remained unclear if these synthetic languages actually resembled human language or if they could be useful for real-world Natural Language Processing (NLP). This paper tackles the missing link between the synthetic protocols of AI agents and the complex, grounded nature of human linguistics.

Methodology: Corpus Transfer and Translation

The authors introduce two primary innovations to connect emergent and natural languages:

  1. Corpus Transfer: Instead of directly transferring the weights of an AI agent to a new task (model transfer), the authors use a trained "speaker" agent to generate a massive dataset—a corpus—of emergent messages grounded in images. They then use this corpus to pre-train a standard Transformer model before fine-tuning it on natural language tasks like Language Modeling (LM) and Image Captioning (IC).
  2. Translation-based Metric: To evaluate how "transferable" an emergent language is, the authors propose a "Translation" metric. They train a model to translate emergent messages into human-written captions for the same images. The higher the translation score (ROUGE-L), the more "human-like" the emergent language's underlying structure is deemed to be.
Overview of the corpus transfer and translation metric framework.

To test this framework, the agents were trained using 2.8M images from the Conceptual Captions dataset. Crucially, the agents did not see the human captions during the game; they only used the raw images to invent their language. Downstream evaluation was performed on Wikipedia subsets for 10 diverse languages (for LM) and the MS-COCO dataset (for IC).

Key Insights and Results

Emergent Language as a "Warm Start"

Even though the AI agents invented their own numerical tokens (e.g., "49 28 39..."), the resulting sequences contain structural patterns that help a Transformer learn the "shape" of human language.

  • Language Modeling: Pre-training on an emergent corpus with just 2 million tokens reduced model perplexity by an average of 24.6% across ten natural languages compared to training from scratch.
  • Image Captioning: EC pre-training provided roughly half the benefit of pre-training on actual human captions. This is remarkable considering the EC process requires zero human labels—only raw images.

The "Simplicity" Advantage in Low-Resource Settings

In extremely low-resource settings (2 million tokens), pre-training on emergent language actually outperformed pre-training on natural language (Spanish Wikipedia). The authors hypothesize that emergent language, generated by simple GRU agents, is "cleaner" and lacks the overwhelming noise and complexity of Wikipedia. This makes it a better "starter" language—much like how children learn from simplified "parentese."

Language modeling perplexity across 10 languages, comparing emergent (ec) and natural (es) pre-training at different source sizes.

However, there are scaling limits. As the pre-training corpus grows to 30M tokens, the benefit of natural language eventually overtakes the emergent language, indicating a plateau in the utility of current EC protocols at scale.

The Flaw in Topographic Similarity

A major revelation in the paper is the failure of topographic similarity—the gold-standard metric in EC for years. It showed almost zero correlation ($\rho = 0.003$) with downstream performance. In contrast, the authors' new translation metric was highly predictive ($\rho = 0.83$). This suggests that simple attribute disentanglement (the goal of topographic similarity) is not what makes a language human-like or useful for NLP.

Limitations and Future Work

While corpus transfer proves that emergent languages have real-world NLP utility, challenges remain:

  • Agent Complexity: The emergent languages were generated by relatively simple, one-layer GRU agents. Using more sophisticated agents or more complex games might result in emergent languages that scale better and capture deeper nuances of human grammar.
  • The Scaling Gap: Future work needs to investigate how to "evolve" these languages to maintain their performance lead over natural language corpora as data volume increases.
  • Task Breadth: Testing corpus transfer on more complex reasoning tasks (like Natural Language Inference or Question Answering) would further validate the "human-likeness" of these AI protocols.

Summary updated

Abstract

The study of language emergence aims to understand how human languages are shaped by perceptual grounding and communicative intent. Computational approaches to emergent communication (EC) predominantly consider referential games in limited domains and analyze the learned protocol within the game framework. As a result, it remains unclear how the emergent languages from these settings connect to natural languages or provide benefits in real-world language processing tasks, where statistical models trained on large text corpora dominate. In this work, we propose a novel way to establish such a link by corpus transfer, i.e. pretraining on a corpus of emergent language for downstream natural language tasks, which is in contrast to prior work that directly transfers speaker and listener parameters. Our approach showcases non-trivial transfer benefits for two different tasks – language modeling and image captioning. For example, in a low-resource setup (modeling 2 million natural language tokens), pre-training on an emergent language corpus with just 2 million tokens reduces model perplexity by 24.6% on average across ten natural languages. We also introduce a novel metric to predict the transferability of an emergent language by translating emergent messages to natural language captions grounded on the same images. We find that our translation-based metric highly correlates with the downstream performance on modeling natural languages (for instance $\rho = 0.83$ on Hebrew), while topographic similarity, a popular metric in previous works, shows surprisingly low correlation ($\rho = 0.003$), hinting that simple properties like attribute disentanglement from synthetic domains might not capture the full complexities of natural language. Our findings also indicate potential benefits of moving language emergence forward with natural language resources and models.
BibTeX
@inproceedings{yao2022linking,
  author = {Shunyu Yao and Mo Yu and Yang Zhang and Karthik R. Narasimhan and Joshua B. Tenenbaum and Chuang Gan},
  title = {{Linking Emergent and Natural Languages via Corpus Transfer}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=E-hf5aN5Ig},
}
Figure