Linking Emergent and Natural Languages via Corpus Transfer

Shunyu Yao, Mo Yu, Yang Zhang, Karthik R. Narasimhan, Joshua B. Tenenbaum, Chuang Gan

Introduction

In the field of Emergent Communication (EC), researchers study how artificial agents develop their own protocols to solve communicative tasks, such as identifying or describing images. While these emergent languages often exhibit structural properties, they have historically been confined to synthetic, "toy" environments. A persistent challenge in the field is determining whether these protocols actually resemble human natural language (NL) in any functional way.

Previous attempts to link EC and NL focused on "model transfer"—the process of moving neural weights from an agent to a downstream model. However, this approach is often brittle and constrained by specific agent architectures. This paper proposes a more flexible and robust link: corpus transfer. By treating the output of emergent agents as a pre-training corpus for standard NLP models, the authors demonstrate that emergent communication can provide a significant "head start" for learning real-world human languages.

The Corpus Transfer Framework

The researchers developed a three-stage framework to test the utility of emergent protocols:

  1. Language Emergence: A "Speaker" and "Listener" agent are trained in a referential game using natural images from the Conceptual Captions dataset. The Speaker generates a sequence of discrete tokens to describe an image, and the Listener must identify that image from a distractor set.
  2. Corpus Generation: Once the agents achieve proficiency, the Speaker is used to generate a massive corpus of emergent messages based on millions of images.
  3. Corpus Transfer: A standard Transformer model is pre-trained on this emergent corpus. This model is then fine-tuned on two distinct natural language tasks: Language Modeling (LM) across 10 diverse languages (ranging from Basque to Urdu) and Image Captioning (IC) using the MS-COCO dataset.

Redefining Evaluation: The Translation Metric

For years, the EC community has relied on "topographic similarity"—a measure of how much the distance between messages correlates with the distance between the inputs they describe—as the gold standard for evaluating emergent languages. This paper presents a significant finding: topographic similarity has almost zero correlation ($\rho = 0.003$) with a model's ability to transfer to natural language.

To replace this, the authors introduce a Translation Metric. By training a model to translate emergent messages into English captions for the same images, they use the ease of translation (measured via ROUGE-L) as a proxy for the language's quality. This metric proved to be a far superior predictor of downstream performance, showing a high correlation ($\rho = 0.83$) with transfer success.

Key Results and Insights

1. Significant Performance Gains

Pre-training on just 2 million emergent tokens reduced test perplexity by an average of 24.6% across ten natural languages compared to training from scratch. In image captioning tasks with limited data (5k samples), EC pre-training improved the BLEU-4 score from 14.1 to 15.3.

2. The Power of Structure

A crucial insight from this work is that the structure of the language matters more than its specific semantics. Ablation studies showed that if the word order of emergent messages was shuffled or if a random speaker was used, the transfer benefits vanished. This suggests that the learned syntax and communicative intent grounded in visual data provide the foundational structure necessary for a Transformer to learn human grammar more efficiently.

3. Success in Low-Resource Settings

The benefits of EC pre-training are most pronounced when natural language data is scarce. In these low-resource scenarios, pre-training on an emergent corpus (ec) consistently outperformed pre-training on a small amount of unrelated natural language, such as Spanish Wikipedia (es).

Limitations and Future Directions

While the results are promising, the authors note a "scaling ceiling." As the amount of available natural language data increases, the relative benefits of EC pre-training diminish. Current EC agents still fail to replicate the full complexity of natural language found in massive corpora like Wikipedia. Furthermore, the multi-stage pipeline is computationally intensive compared to direct evaluation methods.

However, this framework provides a new "fitness function" for the evolution of AI agents. By using transferability to natural language as a metric, researchers can evolve agents whose communication protocols are increasingly aligned with human-like understanding. This paves the way for AI that can bridge the gap between synthetic reasoning and real-world communication.

Summary updated

BibTeX
@inproceedings{yao2022linking,
  author = {Shunyu Yao and Mo Yu and Yang Zhang and Karthik R. Narasimhan and Joshua B. Tenenbaum and Chuang Gan},
  title = {{Linking Emergent and Natural Languages via Corpus Transfer}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=E-hf5aN5Ig},
}
Figure