Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication

Lukasz Kucinski, Tomasz Korbak, Pawel Kolodziej, Piotr Milos

Introduction

Compositional communication—the ability to construct complex messages from simpler, meaningful subparts—is a hallmark of human language and a critical goal for emergent communication in AI. While often assumed to emerge naturally from task complexity, this paper demonstrates that compositionality requires specific inductive biases and that noise in the communication channel serves as a vital catalyst for its development.

The Theoretical Necessity of Bias

The authors present a discrete "impossibility theorem" (Theorem 1), proving that in a perfectly balanced dataset, a compositional mapping is mathematically indistinguishable from a permuted, non-compositional one without an a priori bias. This challenges the notion that compositionality can be learned in a purely unsupervised manner.

Comparison of compositional and non-compositional languages.

As illustrated in Figure 2, different languages (e.g., $\ell, \ell_\pi$) can map the same objects to the same symbols but differ fundamentally in their structural alignment with features like color and shape. Without grounding priors—either in the model architecture or the data distribution—agents have no mathematical basis to prefer a compositional structure over a holistic one.

Noise as a Catalyst for Structure

The core proposal introduces a "noisy channel" between the Sender and Receiver. By randomly flipping symbols with a probability $\epsilon$, the environment penalizes holistic languages where a single error renders a message entirely meaningless. In contrast, compositional languages are more robust; an error in one symbol might only corrupt one feature (e.g., shape), allowing the Receiver to still correctly identify others (e.g., color). This "partial credit" pressure drives agents toward structured protocols. The authors identify a "Goldilocks" zone (around $\epsilon = 0.1$) where compositionality peaks before excessive noise degrades overall communication accuracy.

Experimental Methodology

The study employs signaling games across several synthetic vision datasets to validate these theoretical insights.

Examples from Shapes3d, Obverter, and scrambled variants.

Datasets include Shapes3d (3D objects with independent features like floor color and object scale) and the Obverter dataset (2D shapes in various colors). To test the role of visual priors, the authors also utilized scrambled variants (bottom row of Figure 3), where images are tiled to disrupt global shape recognition, making it harder for standard neural networks to leverage inherent visual structure.

Key Results and Discussion

Empirical results on the Shapes3d dataset show that adding a noise level of 0.1 significantly boosts compositionality, increasing Topographic Similarity from 0.79 to 0.87. Key findings include:

  • Architectural Impact: The strongest bias toward compositionality was observed when using a "two-headed" network output, explicitly forcing the model to predict features separately.
  • Generalization: Agents trained with noise were significantly better at "zero-shot" communication—describing feature combinations never seen during training—after minimal fine-tuning.
  • Accuracy vs. Structure: While noise improves structural metrics like positional disentanglement, it creates a trade-off with absolute communication accuracy if the noise level is too high.

Conclusion

This research highlights that compositionality is not an inevitable outcome of complex tasks but a result of specific environmental pressures and architectural priors. While the study uses simplified supervised setups, it provides a rigorous foundation for understanding language emergence. Future work will explore these dynamics in complex Reinforcement Learning scenarios and investigate the boundary between simple compositionality and complex error-correcting codes.

Summary updated

Abstract

Communication is compositional if complex signals can be represented as a combination of simpler subparts. In this paper, we theoretically show that inductive biases on both the training framework and the data are needed to develop a compositional communication. Moreover, we prove that compositionality spontaneously arises in the signaling games, where agents communicate over a noisy channel. We experimentally confirm that a range of noise levels, which depends on the model and the data, indeed promotes compositionality. Finally, we provide a comprehensive study of this dependence and report results in terms of recently studied compositionality metrics: topographical similarity, conflict count, and context independence.
BibTeX
@inproceedings{kucinski2021catalytic,
  author = {Lukasz Kucinski and Tomasz Korbak and Pawel Kolodziej and Piotr Milos},
  title = {{Catalytic Role Of Noise And Necessity Of Inductive Biases In The Emergence Of Compositional Communication}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=FAm_H6ZcuMv},
}
Figure