Introduction
Compositional communication—the ability to construct complex messages from simpler, meaningful subparts—is a hallmark of human language and a critical goal for emergent communication in AI. While often assumed to emerge naturally from task complexity, this paper demonstrates that compositionality requires specific inductive biases and that noise in the communication channel serves as a vital catalyst for its development.
The Theoretical Necessity of Bias
The authors present a discrete "impossibility theorem" (Theorem 1), proving that in a perfectly balanced dataset, a compositional mapping is mathematically indistinguishable from a permuted, non-compositional one without an a priori bias. This challenges the notion that compositionality can be learned in a purely unsupervised manner.
As illustrated in Figure 2, different languages (e.g., $\ell, \ell_\pi$) can map the same objects to the same symbols but differ fundamentally in their structural alignment with features like color and shape. Without grounding priors—either in the model architecture or the data distribution—agents have no mathematical basis to prefer a compositional structure over a holistic one.
Noise as a Catalyst for Structure
The core proposal introduces a "noisy channel" between the Sender and Receiver. By randomly flipping symbols with a probability $\epsilon$, the environment penalizes holistic languages where a single error renders a message entirely meaningless. In contrast, compositional languages are more robust; an error in one symbol might only corrupt one feature (e.g., shape), allowing the Receiver to still correctly identify others (e.g., color). This "partial credit" pressure drives agents toward structured protocols. The authors identify a "Goldilocks" zone (around $\epsilon = 0.1$) where compositionality peaks before excessive noise degrades overall communication accuracy.
Experimental Methodology
The study employs signaling games across several synthetic vision datasets to validate these theoretical insights.
Datasets include Shapes3d (3D objects with independent features like floor color and object scale) and the Obverter dataset (2D shapes in various colors). To test the role of visual priors, the authors also utilized scrambled variants (bottom row of Figure 3), where images are tiled to disrupt global shape recognition, making it harder for standard neural networks to leverage inherent visual structure.
Key Results and Discussion
Empirical results on the Shapes3d dataset show that adding a noise level of 0.1 significantly boosts compositionality, increasing Topographic Similarity from 0.79 to 0.87. Key findings include:
- Architectural Impact: The strongest bias toward compositionality was observed when using a "two-headed" network output, explicitly forcing the model to predict features separately.
- Generalization: Agents trained with noise were significantly better at "zero-shot" communication—describing feature combinations never seen during training—after minimal fine-tuning.
- Accuracy vs. Structure: While noise improves structural metrics like positional disentanglement, it creates a trade-off with absolute communication accuracy if the noise level is too high.
Conclusion
This research highlights that compositionality is not an inevitable outcome of complex tasks but a result of specific environmental pressures and architectural priors. While the study uses simplified supervised setups, it provides a rigorous foundation for understanding language emergence. Future work will explore these dynamics in complex Reinforcement Learning scenarios and investigate the boundary between simple compositionality and complex error-correcting codes.
Summary updated