Generating Bilingual Pragmatic Color References

Will Monroe, Jennifer Hu, Andrew Jong, Christopher Potts

Abstract

This research explores the intersection of pragmatics and bilingualism by training a neural agent to play a "color reference game" in both English and Mandarin Chinese. We demonstrate that a single model can learn to be more pragmatically informative by leveraging cross-lingual regularities, even in the absence of parallel translation data. Our findings indicate that bilingual training significantly improves performance in lower-resource scenarios and allows for the discovery of cross-lingual synonyms through grounded experience.

Introduction

In human communication, pragmatics dictates that we use language to distinguish objects from their context. For instance, when identifying a specific blue mug among others, a speaker might say "the bright one" rather than just "the blue mug" to resolve ambiguity. While these pragmatic pressures are universal, their linguistic expression varies based on a language's syntax and vocabulary. This study addresses whether English and Chinese speakers react to contextual difficulty similarly and if a machine learning model can transfer pragmatic skills from a high-resource language (English) to a lower-resource one (Chinese) without explicit translation pairs.

Experimental Framework

The researchers employed a Reference Game framework where a "speaker" model must describe a target color among two distractors so that a "listener" can identify it.

Example trials from the color reference game in Mandarin Chinese.

Model Architecture

The model utilizes a sequence-to-sequence LSTM encoder-decoder. To handle bilingualism, a simple "language flag" is added to the input. Crucially, the model shares hidden layers for both languages, forcing it to map colors into a shared conceptual space before generating descriptions.

Datasets and Contexts

The study used an existing English dataset (~50,000 descriptions) and a newly collected Mandarin Chinese dataset (5,774 messages). Trials were categorized by difficulty:

  • Far: Target is easily distinguishable.
  • Split: One distractor is similar to the target.
  • Close: All colors are highly similar.

Pragmatic Universals and Insights

Analysis of human data revealed shared pragmatic patterns across both languages. As the context becomes more difficult (the "Close" condition), speakers in both English and Chinese increase their verbosity, use more negations, and employ more specific terminology.

Human verbosity increases as the target color becomes harder to distinguish from distractors.

Key Findings

  • Performance Gains: The bilingual model achieved a pragmatic informativeness score of 72.89% on Chinese data, compared to 67.75% for the monolingual version. This suggests bilingual training is particularly beneficial for low-resource scenarios.
  • Zero-Shot Lexicon Induction: By analyzing the model's output layer vectors, researchers could identify synonyms (e.g., mapping "blue" to lánsè) without any parallel text or dictionary.
  • Semantic Integrity: Despite the shared architecture, the model maintained language-specific nuances, such as English's specific use of "teal" versus more general Chinese visual metaphors.

Conclusion

This work demonstrates that pragmatic competence can be transferred across languages through grounded multi-modal learning. The bilingual model not only outperformed monolingual baselines in Chinese but also exhibited more human-like behavior by adjusting its verbosity based on contextual difficulty. Future research may explore these dynamics in languages with more divergent color systems to further test the limits of shared semantic representations.

Summary updated

Abstract

Contextual influences on language often exhibit substantial cross-lingual regularities; for example, we are more verbose in situations that require finer distinctions. However, these regularities are sometimes obscured by semantic and syntactic differences. Using a newly-collected dataset of color reference games in Mandarin Chinese (which we release to the public), we confirm that a variety of constructions display the same sensitivity to contextual difficulty in Chinese and English. We then show that a neural speaker agent trained on bilingual data with a simple multitask learning approach displays more human-like patterns of context dependence and is more pragmatically informative than its monolingual Chinese counterpart. Moreover, this is not at the expense of language-specific semantic understanding: the resulting speaker model learns the different basic color term systems of English and Chinese (with noteworthy cross-lingual influences), and it can identify synonyms between the two languages using vector analogy operations on its output layer, despite having no exposure to parallel data.
BibTeX
@inproceedings{monroe2018generating,
  author = {Will Monroe and Jennifer Hu and Andrew Jong and Christopher Potts},
  title = {{Generating Bilingual Pragmatic Color References}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-1196},
}
Figure