A Noisy-Channel Model of Human Sentence Comprehension under Uncertain Input

Roger LevyEMNLP 2008 · OpenReview

Introduction

Traditional psycholinguistic models typically assume "clean input," where the brain processes a perfectly perceived sequence of words. However, real-world communication is fraught with noise, including environmental sounds, speech errors, and typos. Roger Levy (2008) proposes a "noisy-channel" model, suggesting that human sentence comprehension is a rational Bayesian inference process that treats linguistic input as potentially corrupted. This framework explains why humans often "hallucinate" likely structures and experience processing delays in specific ambiguous contexts.

The Noisy-Channel Framework

The core of the model is the goal to recover the most likely intended string $w$ given a perceived, potentially noisy input $I$. This is formalized as:

$$P(w|I) \propto P(I|w)P(w)$$

where $P(w)$ is the Prior (the listener's knowledge of language) and $P(I|w)$ is the Noise Model.

The prior is represented by a Probabilistic Context-Free Grammar (PCFG) with probabilities derived from the Brown Corpus. The noise model is a kernel based on Levenshtein distance, accounting for word substitutions, deletions, and insertions. Levy implements this noise model as a weighted Finite-State Automaton (wFSA).

The noise model represented as a weighted Finite-State Automaton (wFSA) for a small vocabulary.

By intersecting the PCFG with the wFSA, the model can calculate the probability of various parses even when the input string itself is grammatically problematic. This suggests that our knowledge of grammar can actually influence our perception of what words were spoken or written.

Measuring Processing Difficulty

To quantify the cognitive load or "surprise" experienced during incremental processing, the paper introduces the Error Identification Signal (EIS). The EIS is defined as the Kullback-Leibler (KL) divergence between the probability distribution of the sentence's prefix before and after encountering a new word. A high EIS indicates that the new word significantly shifts the listener's belief about the intended message, often by making a previously unlikely "near-neighbor" interpretation more probable.

Results and Discussion

The model was tested on sentence structures known to cause human processing difficulties, such as "locally coherent" sentences (e.g., "The coach smiled at the player tossed a frisbee"). In these cases, standard "clean input" models fail to explain why readers slow down at "tossed," as the preceding context should have ruled out its use as a main verb.

Levy's model, however, successfully predicts a spike in the EIS at these critical points. The difficulty arises because "tossed" has many high-probability "near-neighbor" sentences (e.g., "The coach smiled and the player tossed...") that the rational system considers as likely intended messages.

The Error Identification Signal (EIS) for locally coherent vs. incoherent sentences across different noise levels.

Furthermore, the model accounts for global inference errors. In garden-path sentences like "While the man hunted the deer ran into the woods," the model's top parses often include interpretations where "the deer" is the object of "hunted," mirroring common human misinterpretations. These results were robust across various noise parameter settings, indicating that the effects are inherent to the noisy-channel architecture.

Conclusion

The noisy-channel model provides a unified rational explanation for several puzzling phenomena in human language processing. By treating input as uncertain, the model demonstrates that "errors" in human comprehension may actually be the result of a highly efficient system attempting to recover meaning from a noisy world. Future work aims to scale these models to broader grammars and incorporate more sophisticated perceptual noise models.

Summary based on: Levy, R. (2008). A Noisy-Channel Model of Human Sentence Comprehension under Uncertain Input. EMNLP.

Summary updated

Abstract

Roger Levy. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. 2008.
BibTeX
@inproceedings{levy2008noisy,
  author = {Roger Levy},
  title = {{A Noisy-Channel Model of Human Sentence Comprehension under Uncertain Input}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2008},
  url = {https://openreview.net/forum?id=Hy-4VfGOWH},
}
Figure