Communicating artificial neural networks develop efficient color-naming systems

Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, Marco BaroniProceedings of the National Academy of Sciences 2021 · DOI

TL;DR

This paper demonstrates that artificial neural networks trained on a simple color discrimination game develop communication systems remarkably similar to human color-naming languages in terms of efficiency and complexity. Crucially, this efficiency and preference for low complexity are driven by the discrete nature of communication, suggesting it's a general property of discrete communication systems, not unique to human biology.

What Problem Does This Paper Address?

Human languages, despite their vast diversity, exhibit universal constraints, particularly in how words categorize semantic fields. A key question is how these systems evolve to balance the need for accurate communication with the desire to keep languages simple and avoid excessive complexity. This paper investigates this trade-off, specifically in the domain of color naming, by asking whether artificial systems trained on a communication task can spontaneously develop similar efficient and low-complexity naming strategies observed in human languages. The authors aim to understand if these properties are unique to human cognition or if they emerge from fundamental principles of communication.

Methods

The core of the methodology involves training two artificial neural network agents, a "Speaker" and a "Listener," to play a color discrimination game. The Speaker receives a target color and must choose a word from a fixed vocabulary to communicate it to the Listener. The Listener, given the word and a distractor color, must identify the target.

Key techniques include:

  • Deep Learning: Standard feed-forward neural networks are used for both Speaker and Listener.
  • Information Bottleneck (IB) Framework: The authors adopt the IB framework to quantify the accuracy/complexity trade-off. Accuracy is measured by the mutual information between the input (colors) and the output (words), while complexity is measured by the mutual information between the meanings and the words.
  • Discriminative Need Simulation: To mimic variations in human communities, "discriminative need" is simulated by varying the minimum Euclidean distance allowed between target and distractor colors in the CIELAB space. This is controlled by a percentile parameter.
  • Discrete vs. Continuous Communication: The study explicitly contrasts communication through discrete symbols (like words) with continuous message passing. This is achieved using two training methods: REINFORCE (fully discrete) and Gumbel-Softmax (allowing for a continuous relaxation during training).

Datasets

  • World Color Survey (WCS): This dataset, containing color chip names from 330 colors across 110 non-industrialized languages, is used as a benchmark for human color-naming systems.
  • English Language Approximation: English color naming data is approximated based on existing literature (Zaslavsky et al.).
  • No New Benchmarks: The paper does not introduce new datasets but uses existing ones to compare emergent NN systems with human language data.

Key Insights and Ideas

  • Emergent Efficiency: Artificial neural networks, when trained on a simple communication game, spontaneously develop color-naming systems that are highly efficient, lying close to the theoretical Information Bottleneck limit, just like human languages. This suggests efficiency is not a uniquely human trait but a general outcome of optimizing communication.
  • Preference for Low Complexity: Both human and emergent NN systems show a strong preference for low-complexity solutions, even if it means sacrificing perfect accuracy. This "low-complexity bias" is a striking commonality.
  • Discreteness as a Bottleneck: The paper's most significant finding is that the discrete nature of communication is the crucial factor driving both efficiency and low complexity. When communication is allowed to be continuous, the emergent systems become more complex and less efficient, deviating from human language patterns.
  • Universal Principles: The results suggest that efficient semantic categorization is a general property of discrete communication systems, not necessarily tied to specific biological constraints of humans.

Key Results

  • Accuracy/Complexity Match: The distribution of emergent NN color-naming systems on the accuracy/complexity plane closely matches that of human languages, including both WCS languages and English.
  • Near-Optimal Efficiency: NN systems achieve near-optimal efficiency, with their inefficiency scores falling within the human range.
  • Narrow Complexity Range: Both human and NN systems occupy a narrow segment of the theoretical IB curve, tilted towards lower complexity.
  • Discriminative Need Impact: Varying the "discriminative need" (how finely colors must be distinguished) affects the complexity of NN systems (higher need leads to more complex systems), but not their efficiency. This mirrors observations in human language adaptation.
  • Discrete Communication is Key: Training with discrete communication (REINFORCE) resulted in significantly less complex systems compared to training with continuous relaxation (Gumbel-Softmax). Smoother communication channels led to more complex and less efficient systems.

Limitations and Future Work

  • Two-Word System Avoidance: The paper notes that both human and NN systems avoid two-word codes, but offers no definitive explanation for this phenomenon.
  • Agent Capacity: While agent capacity (network size) was explored, its impact was primarily observed in discrete communication settings, reinforcing the importance of the channel's discreteness.
  • Future Work: The authors suggest designing experiments with human participants to explore the causal relationship between discreteness and communicative efficiency, and how a transition from continuous to discrete communication might shape semantic categorization. They also propose investigating how this principle applies to other semantic domains beyond color.

Summary updated

Abstract

Words categorize the semantic fields they refer to in ways that maximize communication accuracy while minimizing complexity. Focusing on the well-studied color domain, we show that artificial neural networks trained with deep-learning techniques to play a discrimination game develop communication systems whose distribution on the accuracy/complexity plane closely matches that of human languages. The observed variation among emergent color-naming systems is explained by different degrees of discriminative need, of the sort that might also characterize different human communities. Like human languages, emergent systems show a preference for relatively low-complexity solutions, even at the cost of imperfect communication. We demonstrate next that the nature of the emergent systems crucially depends on communication being discrete (as is human word usage). When continuous message passing is allowed, emergent systems become more complex and eventually less efficient. Our study suggests that efficient semantic categorization is a general property of discrete communication systems, not limited to human language. It suggests moreover that it is exactly the discrete nature of such systems that, acting as a bottleneck, pushes them toward low complexity and optimal efficiency.
BibTeX
@inproceedings{chaabouni2021communicating,
  author = {Rahma Chaabouni and Eugene Kharitonov and Emmanuel Dupoux and Marco Baroni},
  title = {{Communicating artificial neural networks develop efficient color-naming systems}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2021},
  doi = {10.1073/pnas.2016569118},
}
Figure