TL;DR
This paper introduces a novel computational model that simulates how humans process noisy or anomalous language by combining a language model with an error model and a resource-rational inference algorithm. The model demonstrates that computational constraints, like the number of processing "particles," influence the model's ability to infer intended meanings, mirroring human-like trade-offs between accuracy and effort.
What Problem Does This Paper Address?
Human language comprehension is remarkably robust, allowing us to understand utterances even when they contain errors or are otherwise anomalous. This raises a fundamental question: how do humans achieve this feat in real-time, especially when faced with an infinite space of potential interpretations and limited cognitive resources? Existing language models, primarily trained on predicting the next word, often struggle with these "noisy-channel" scenarios. This paper aims to bridge this gap by developing a computational model that explicitly accounts for both the probability of intended language (the prior) and the likelihood of errors occurring (the error model), all within a resource-constrained inference framework.
Methods
The core of the model is a noisy-channel framework where comprehension is framed as inferring an intended sentence ($s$) given a noisy utterance ($u$). This is achieved using Bayes' Rule: $P(s|u) = P(u|s)P(s) / \sum_{s'} P(u|s')P(s')$.
The key innovations lie in:
- Generative Model: This comprises:
- An autoregressive Language Model (LM) (specifically GPT-2) that provides the prior probability $P(s)$ of intended sentences. To ensure a manageable vocabulary, a "locally constrained decoding" (LCD) method is used.
- An Error Model that simulates various types of errors (normal production, insertion, skip, form-based, semantic, and morphological substitutions) that can transform an intended sentence into a noisy utterance.
- Inference Algorithm: Sequential Monte Carlo (SMC) is employed for incremental, word-by-word inference. To address the intractability of exact inference and mimic human reanalysis capabilities, SMC is augmented with:
- Rejuvenation Proposals: These are mechanisms (e.g., Form-based, Semantic, Insertion/Deletion) that allow particles (hypotheses about the sentence state) to be modified in light of new observations, enabling reanalysis of earlier parts of the sentence.
- Resource Rationality: The model explicitly tests the impact of computational constraints, primarily by varying the number of particles ($K$) in the SMC algorithm and the extent of rejuvenation (e.g., number of iterations, lookback parameter $\lambda$).
Datasets
The paper utilizes existing datasets for evaluation:
- Ryskin et al. (2021): Used in Experiment 1 to evaluate the model's ability to handle anomalous words (neighboring errors vs. unrelated errors) and its performance across different particle counts.
- Qian and Levy (2023): Used in Experiment 2 to assess the model's verb-edit preferences in sentences with subject-verb agreement errors, comparing its reanalysis capabilities with human behavior.
No new benchmarks were introduced.
Key Insights and Ideas
- Computational Constraints Shape Inference: The number of particles ($K$) in the SMC algorithm directly impacts the model's ability to discover low-surprisal interpretations for anomalous sentences. More particles lead to better performance, especially for "recoverable" errors.
- Reanalysis is Crucial for Human-like Behavior: Augmenting purely incremental processing with rejuvenation (reanalysis) significantly improves the model's fit to human verb-edit preferences in agreement error sentences.
- Distinguishing Error Types: The model shows a dissociation between how it handles "recoverable" errors (like typos or semantic neighbors) and "unrelated" errors. For recoverable errors, increased computation leads to lower surprisal, mirroring human behavior, whereas unrelated errors remain high-surprisal.
- Algorithmic Trade-offs: The model provides a concrete instantiation of "resource rationality," demonstrating how algorithmic choices (particle count, rejuvenation strategy) lead to a trade-off between inference accuracy and computational cost, mimicking human cognitive effort.
Key Results
- Experiment 1 (Incremental Inference): For recoverable errors (Neighbor, Ungrammatical conditions), noisy-channel surprisal became lower than baseline LM surprisal with sufficient particles (more than 8). For $K=128$, noisy-channel surprisal was 1-2 bits lower than baseline. Unrelated errors did not show this benefit.
- Experiment 2 (Reanalysis): Second-pass rejuvenation with 2 iterations and conditional rejuvenation with a lookback parameter $\lambda > 2$ improved the model's correlation with human verb-edit preferences in agreement error sentences. The best model correlation was 0.48 for second-pass rejuvenation and 0.34 for conditional rejuvenation.
Limitations and Future Work
- Error Model Expressiveness: The current error model omits certain error types (e.g., word exchanges) and has limitations in modeling skips (only one word). It also assumes errors always result in real words, not non-words.
- Language Model Specificity: The model's sensitivity to the choice of LM and its vocabulary restriction (via LCD) warrants further investigation.
- Calibration: The probabilities assigned to error transformations are not calibrated to actual production statistics.
- Rejuvenation Heuristics: The rejuvenation proposals are heuristic and might not be optimal. Future work could explore more targeted error localization.
- Future Directions:
- Investigate dynamic processing policies where computational resources adapt to inference difficulty.
- Explore how experimental manipulations (e.g., speeded judgments, incentives) elicit human behavioral profiles matching varying computational resources.
- Extend the error model to include non-word errors and word exchanges.
- Adapt the model to morphologically complex languages.
- Explore more sophisticated rejuvenation strategies.
- Investigate the role of different LMs and prompts.
Summary updated