Ensemble Kalman filter for uncertainty in human language comprehension

Diksha Bhandari, Alessandro Lopopolo, Milena Rabovsky, Sebastian ReicharXiv 2025 · DOI · arXiv

Abstract

Traditional neural networks for language processing are often overconfident in their predictions, even when processing linguistically anomalous inputs. This paper introduces a Bayesian framework utilizing an Ensemble Kalman Filter (EnKF) to help models quantify uncertainty. By moving from deterministic point-estimates to a probabilistic approach, the authors demonstrate that models can better mimic human-like "double-takes" when encountering confusing or nonsensical sentences, such as reversal anomalies.

Introduction

While humans are remarkably good at language, we often experience cognitive conflict when a sentence violates common sense despite being syntactically correct—for example, reversal anomalies like "The dog was bitten by the man" or "The eggs would eat breakfast." In these cases, the human brain juggles a conflict between syntax (structure) and semantics (plausibility).

Current Artificial Neural Networks (ANNs), specifically the Sentence Gestalt (SG) model, tend to be deterministic. When faced with these anomalies, they often confidently select the most plausible meaning based on training data, ignoring the syntactic cues that suggest an anomaly. They lack a mechanism to represent doubt. This research aims to bridge this gap by implementing a probabilistic Bayesian approach to handle linguistic uncertainty.

The Sentence Gestalt Architecture

The researchers modified the existing Sentence Gestalt (SG) model, a recurrent neural network that builds a mental "gestalt" or summary of a sentence word-by-word.

The Sentence Gestalt (SG) model architecture, featuring an Update Network for processing sentences and a Query Network for extracting event roles.

As shown in Figure 1, the model consists of an Update Network that processes the sentence and a Query Network that uses a probe to extract semantic roles such as Agent, Action, and Patient.

Methodology: Bayesian Inference via EnKF

To introduce uncertainty, the authors moved away from standard Maximum Likelihood Estimation (MLE), which produces a single "best guess" for model weights. Instead, they implemented Bayesian inference on the model's output layer. Because calculating the true Bayesian posterior is computationally intractable for neural networks, they used a sophisticated approximation technique: the Ensemble Kalman Filter (EnKF) Sampler.

Integration of EnKF-based methods into the SG model to update the ensemble of weights for the output layer based on feature maps and target labels.

As illustrated in Figure 2, this method treats the search for model weights as an interacting particle system. An "ensemble" of different weight configurations (particles) evolves over time toward the most likely distribution. Key components of this implementation include:

  • Dropout Regularization: Integrated into the EnKF process to prevent the ensemble from collapsing into a single subspace, ensuring a diverse representation of possible interpretations.
  • Two-Stage Training: The model is first trained using standard MLE to learn basic language features, followed by the application of the Bayesian EnKF specifically to the final query layer.

Experimental Results

The study utilized a synthetically generated corpus of 10,000 sentences to compare the MLE and Bayesian models.

  • Congruent Sentences: Both models performed similarly well on sensible sentences, showing high activation (confidence) for correct roles.
  • Reversal Anomalies: The MLE model remained highly confident in the semantically plausible but syntactically wrong interpretation. In contrast, the Bayesian model showed a statistically significant shift (p < 0.0001). It exhibited much lower confidence in the "common sense" answer and higher activation for the "syntactically correct" answer compared to the MLE version.

Conclusion

The Bayesian EnKF approach more closely mirrors "good-enough" human processing, where we might lean toward a plausible meaning but remain aware of conflicting sentence structures. The spread in activation serves as a direct mathematical proxy for cognitive uncertainty. Future work will explore applying this framework to the entire network and scaling it to more complex linguistic structures and modern architectures like Transformers.

Summary updated

Abstract

Artificial neural networks (ANNs) are widely used in modeling sentence processing but often exhibit deterministic behavior, contrasting with human sentence comprehension, which manages uncertainty during ambiguous or unexpected inputs. This is exemplified by reversal anomalies-sentences with unexpected role reversals that challenge syntax and semantics-highlighting the limitations of traditional ANN models, such as the Sentence Gestalt (SG) Model. To address these limitations, we propose a Bayesian framework for sentence comprehension, applying an extension of the ensemble Kalman filter (EnKF) for Bayesian inference to quantify uncertainty. By framing language comprehension as a Bayesian inverse problem, this approach enhances the SG model's ability to reflect human sentence processing with respect to the representation of uncertainty. Numerical experiments and comparisons with maximum likelihood estimation (MLE) demonstrate that Bayesian methods improve uncertainty representation, enabling the model to better approximate human cognitive processing when dealing with linguistic ambiguities.
BibTeX
@misc{bhandari2025ensemble,
  author = {Diksha Bhandari and Alessandro Lopopolo and Milena Rabovsky and Sebastian Reich},
  title = {{Ensemble Kalman filter for uncertainty in human language comprehension}},
  year = {2025},
  eprint = {2505.02590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.02590},
}
Figure