The Generalization Puzzle
In the field of emergent communication, researchers train "Sender" and "Receiver" neural networks to communicate about objects or concepts using discrete signals. A long-standing puzzle in this domain is that these models often achieve high accuracy on unseen combinations—demonstrating robust out-of-distribution (O.O.D.) generalization—yet they frequently fail standard "compositionality tests" like Topographic Similarity.
If compositionality is the engine of generalization, how are these models generalizing without it? Some researchers have suggested that neural networks might be finding "non-compositional" ways to generalize. However, this paper argues that our definitions of compositionality have been too narrow, conflating it with strict regularity. The authors demonstrate that emergent languages, much like human languages, can be highly compositional while still being "messy" and variable.
Measuring the "Messiness"
To investigate this, the authors implemented a standard reconstruction game (a Lewisian signaling game). A Sender sees a meaning (e.g., a Subject-Verb-Object predicate like "Ollie loves Osgood") and generates a sequence of discrete symbols. The Receiver must then reconstruct the original meaning from those symbols.
Instead of relying solely on traditional metrics that penalize variation, the core contribution of this paper is the introduction of four new measures of linguistic variation. These measures are bounded between 0 (perfectly regular) and 1 (maximally variable):
- Synonymy: Having multiple ways to express the same meaning.
- Homonymy: Using the same symbol to mean different things depending on context.
- Word-Order Freedom: The degree to which semantic roles (Subject, Verb, Object) can appear in different positions in the signal.
- Entanglement: The degree to which different parts of a meaning are "smeared" across the same positions in a signal.
The study uses a synthetic dataset of two-place predicates with 15,625 unique possible meanings. Crucially, the O.O.D. test split uses Maximum Compound Divergence to ensure the model must generalize to novel combinations of atoms it has only seen elsewhere.
Key Insights and Results
Regularity is a "Moving Target"
By using a Rolling Mixed Effects Model, the authors tracked how variation measures correlate with generalization performance throughout training. Early in training, a language's regularity is a strong predictor of how well it will generalize. However, once a language becomes "regular enough" to support communication, this correlation disappears. A language doesn't need to be perfectly regular to be perfectly functional.
Variation Masks Structure
Emergent languages exhibit synonymy and flexible word order, much like natural languages such as Basque or English. Traditional metrics incorrectly label these languages as non-compositional because they penalize this natural variation.
However, the authors show that even the most "variable" models in their study still showed clear underlying structure. For instance, verbs might appear mostly at the start of a signal and objects at the end. This proves that variation and compositionality are not mutually exclusive. All models achieved high O.O.D. accuracy (between 72.86% and 82.13%), confirming they were indeed learning compositional rules, even with synonymy scores hovering around 0.46–0.63 (where an "ideal" rigid language is 0.0).
The Capacity-Regularity Link
In an effort to decrease the variability of emergent languages, the authors reduced the "brain size" (hidden layer capacity) of the networks. Larger models (800 hidden units) developed significantly more variable languages than smaller models (250 hidden units). By forcing a cognitive bottleneck, the smaller models produced more "disciplined" and regular languages because they lacked the capacity to memorize synonymous variations. This mirrors theories in human linguistics suggesting that cognitive bottlenecks, like limited memory in children, drive the regularization of human language.
Limitations and Future Directions
While these findings bridge a major theoretical gap, the study does have limitations. The SVO predicate task is relatively simple compared to the complexity of natural language or large-scale vision-language tasks. Furthermore, the study focused on GRU-based RNNs; it remains to be seen if Transformers, which handle sequential position differently, would exhibit the same patterns of variation.
Finally, while the paper shows that models only need to be "regular enough" to generalize, it doesn't yet define exactly where that threshold lies. Future work will need to investigate how environmental pressures, such as channel noise, might further interact with model capacity to shape the emergence of regularity.
Summary updated