Searching for the Most Human-like Emergent Language

Brendon Boldt, David R. MortensenEMNLP 2025

TL;DR

This paper tackles the challenge of creating artificial languages that are statistically similar to human languages, a crucial step for using emergent languages as realistic training data for NLP models. By optimizing a signaling game using a novel benchmark called XferBench, the authors discover key hyperparameter settings that lead to more human-like emergent languages and provide evidence that entropy minimization is a fundamental property of these systems.

What Problem Does This Paper Address?

Emergent communication systems, where agents learn to communicate through interaction, hold immense promise for generating synthetic language data. This data could be invaluable for training and evaluating NLP models, especially in low-resource scenarios. However, a significant gap exists in creating emergent languages that are truly human-like in their statistical properties. While many emergent communication systems exist, few have been systematically optimized to mimic the complex patterns found in human language. This paper addresses this by aiming to generate state-of-the-art emergent languages that exhibit high statistical similarity to human language, making them more useful for downstream NLP tasks.

Methods

The core of this research involves a systematic search for optimal hyperparameters within a signaling game environment. The authors employ a Bayesian hyperparameter optimization technique (Tree-structured Parzen Estimator, TPE) to efficiently explore a vast parameter space. The objective function guiding this search is XferBench, a benchmark designed to quantify the similarity of an emergent language to human language by measuring how well a language model pre-trained on the emergent corpus can transfer to downstream human language tasks. To speed up the optimization process, a variant called XferBench-da was used, which evaluates on a single human language (Danish) that showed high correlation with the full XferBench score. The signaling game itself is a discrimination variant with two agents (sender and receiver) interacting in a single round, where messages are sequences of integers.

Datasets

The paper does not introduce new datasets for training. Instead, it focuses on generating emergent language corpora. For evaluation, the authors use:

  • Human languages: A subset of baselines from Boldt and Mortensen (2024b), including Mandarin, Hindi, French, and Arabic.
  • Extant emergent language corpora: Three corpora from ELCC (Emergent Language Corpus Collection) are used for comparison.
  • Search-derived languages: The best-performing emergent languages generated by their hyperparameter search are evaluated.

Key Insights and Ideas

  • Hyperparameter tuning is crucial for human-like emergent languages: The study demonstrates that specific hyperparameter settings, particularly for vocabulary size, message length, and network architecture, significantly impact the human-likeness of emergent languages.
  • Entropy minimization is a fundamental emergent property: The research provides evidence that emergent communication systems naturally minimize entropy with respect to task success, a property not consistently observed in non-emergent synthetic languages. This suggests information-theoretic principles are deeply intertwined with language emergence.
  • XferBench is a powerful, albeit slow, objective: XferBench effectively quantifies human-likeness, but its computational cost necessitates optimizations like XferBench-da for efficient hyperparameter search.
  • Human language similarity is not a simple linear relationship: While some correlations exist between task success, entropy, and XferBench scores, the relationships are complex, indicating that emergent language is not trivially explained by these factors alone.

Key Results

The authors successfully generated emergent languages that achieved state-of-the-art performance on the XferBench benchmark, outperforming existing emergent language corpora and even the Hindi corpus. Specifically, the best emergent languages from their search (XB 1-3) demonstrated a significant step towards human-like language similarity. They also provide concrete recommendations for hyperparameters, suggesting larger vocabulary sizes (around 1000 unique tokens, with a potential of 10000) and longer message lengths than typically used in prior research. The study also confirmed that optimizing solely for entropy, while yielding good results, did not reach the same level of human-likeness as optimizing directly with XferBench.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simplicity of the environment: The signaling game used is relatively basic (single round, fixed sender/receiver, uniform observations), which limits the richness of the communicated information and the resulting languages.
  • Lack of theoretical models: The empirical findings on entropy and task success correlations warrant further theoretical investigation to build robust models and testable hypotheses.
  • Hyperparameter search is "messy": The optimization process is geared towards maximizing performance rather than uncovering generalizable trends, making it harder to extract universal principles.

Future work suggested includes:

  • Introducing more complex signaling games, environments, or neural architectures to generate even more human-like emergent languages.
  • Developing theoretical models to better understand the observed correlations between entropy, task success, and language similarity.
  • Using the generated datasets to empirically test theoretical models of emergent language.

Summary updated

BibTeX
@inproceedings{boldt2025searching,
  author = {Brendon Boldt and David R. Mortensen},
  title = {{Searching for the Most Human-like Emergent Language}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure