Statistical mechanics of lexicon learning in an uncertain and nonuniform world

Rainer Reisenauer, Kenny Smith, Richard A. Blythe

Introduction

Children acquire language with remarkable speed, learning approximately ten words a day despite the inherent "referential uncertainty" of their environment. When a child hears a new word, it is often uttered in a context containing many potential referents. Standard models of cross-situational learning suggest that tracking co-occurrences over time can resolve this ambiguity. However, these models often overlook the non-uniform, Zipfian distribution of meanings in the real world, where certain "distractor" meanings appear far more frequently than others. This paper applies statistical mechanics to show how a simple cognitive heuristic—mutual exclusivity—prevents a catastrophic slowdown in learning.

The Referential Uncertainty Problem

In a world where meanings are uniformly distributed, learning is relatively straightforward. But in reality, some objects (like a table or the floor) are almost always present. These common "confounders" make it difficult to isolate the true meaning of a word through independent observation alone. Recent research indicates that this non-uniformity causes independent learning times to grow super-exponentially as the number of distractors increases, a phenomenon termed the "catastrophe of common confounders."

Modeling Mutual Exclusivity

To address this, the authors model two types of learners: "Noninteracting" and "Interacting." Noninteracting learners track each word-meaning pair independently. Interacting learners utilize the Mutual Exclusivity (ME) constraint—the assumption that each word maps to exactly one unique meaning.

Comparison of noninteracting learning versus interacting learning with the mutual exclusivity constraint.

As shown in Figure 1, the ME constraint couples the learning process. When the meaning of one word (e.g., "square") is identified, it is immediately removed from the set of possible meanings for all other words. This coupling can trigger "learning avalanches," where solving one word clarifies the meanings of several others simultaneously.

Results: Overcoming the Catastrophe

The authors find that the ME heuristic triggers a "dynamical phase transition." Below a critical context size, the lexicon is learned at the physical limit—the time it takes to hear each word at least once. Even above this threshold, the speedup provided by ME is profound.

Lexicon learning time as a function of context size, showing the efficiency of interacting learning compared to independent learning.

Figure 3 illustrates the stark difference in performance. While independent learning time (indicated by the solid blue line for $W=200$) explodes as context size increases, the interacting learner (circles and squares) remains orders of magnitude faster. The difficulty of learning under ME is mathematically linked to the persistence of "candidate loops"—ambiguous cycles of word-meaning mappings that are eventually broken as more data arrives.

Conclusion

By mapping lexicon acquisition to a problem of loop enumeration in statistical mechanics, the authors demonstrate that mutual exclusivity is not just a helpful trick, but a necessary mechanism for rapid language acquisition in a non-uniform world. This heuristic allows children to navigate the "catastrophe of common confounders," maintaining high learning efficiency even in highly uncertain environments.

Summary updated

Abstract

We study the time taken by a language learner to correctly identify the meaning of all words in a lexicon under conditions where many plausible meanings can be inferred whenever a word is uttered. We show that the most basic form of cross-situational learning - whereby information from multiple episodes is combined to eliminate incorrect meanings - can perform badly when words are learned independently and meanings are drawn from a nonuniform distribution. If learners further assume that no two words share a common meaning, we find a phase transition between a maximally-efficient learning regime, where the learning time is reduced to the shortest it can possibly be, and a partially-efficient regime where incorrect candidate meanings for words persist at late times. We obtain exact results for the word-learning process through an equivalence to a statistical mechanical problem of enumerating loops in the space of word-meaning mappings.
BibTeX
@inproceedings{reisenauer2014statistical,
  author = {Rainer Reisenauer and Kenny Smith and Richard A. Blythe},
  title = {{Statistical mechanics of lexicon learning in an uncertain and nonuniform world}},
  booktitle = {APS},
  year = {2014},
}
Figure