Abstract
Mario Giulianelli, Arabella Sinclair, Raquel Fernández. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Introduction
The Uniform Information Density (UID) principle posits that speakers optimize their communication by maintaining a constant rate of information transmission, avoiding "peaks" and "valleys" that might cause processing bottlenecks for listeners. While well-established in monologues and written text, the applicability of UID to the "messy" reality of interactive, task-oriented dialogue—characterized by multiple speakers, grounding acts, and shifting contexts—remains an open question. This study investigates whether UID holds in such settings and at what structural granularity it emerges.
Methodology
To measure information content, the researchers estimated surprisal using GPT-2, fine-tuned on specific dialogue corpora. Unlike traditional $n$-gram models, Transformer-based architectures capture long-range dependencies essential for dialogue context. The core hypothesis is that as a conversation progresses and shared context increases, words become more predictable. To maintain uniform density, speakers must increase the "new" information per word. Thus, a positive correlation between turn position and information content serves as evidence for UID.
The study analyzed three datasets:
- MapTask (MT): Spoken navigation dialogues between an "Instruction Giver" and a "Follower."
- PhotoBook (PB): Written collaborative games focused on identifying shared images.
- Penn Treebank (WSJ): A news text baseline used for validation.
Results: MapTask and Role Asymmetry
In the MapTask corpus, UID was not observed at the global dialogue level. However, a significant effect emerged when analyzing transactions—coherent sub-tasks like reaching a specific landmark.
A critical finding was the impact of speaker roles. In these asymmetric tasks, Instruction Givers followed the UID principle, whereas Followers did not. This discrepancy is attributed to the nature of Follower turns, which often consist of low-information grounding acts (e.g., "okay," "right") that do not follow standard information-loading patterns.
Results: PhotoBook and Referential Compression
In the PhotoBook corpus, a strong positive effect of turn position on information content was found across the entire game, suggesting that participants optimize their communication strategy over the long term as they build "conceptual pacts."
The strongest evidence for UID appeared in reference chains. As participants repeatedly referred to the same image, the length of their descriptions decreased (e.g., from "man eating a slice of pizza" to "pizza eater"), but the information density per word rose. This demonstrates that referential reduction is a form of information compression, allowing speakers to maintain efficiency as common ground grows.
Conclusion
This research demonstrates that UID is not a blanket rule for entire conversations but emerges within topically coherent units like transactions and reference chains. The findings suggest that human speakers are "rational" information processors who adapt their density based on role and context. These insights have significant implications for Natural Language Generation (NLG), suggesting that models should be trained to produce human-like, efficient information profiles by accounting for relevant contextual units.
Summary updated