Speaker commitment—the degree to which a speaker believes an event mentioned in their speech is true, false, or uncertain—is a fundamental component of natural language understanding. For tasks like information extraction or question answering, identifying an event is only half the battle; a system must also determine if the speaker actually commits to that event's truth.
Consider the difference between "I know that Florence is packed" (certainty of truth) and "I don't think that Florence is packed" (certainty of falsehood). In more complex cases like "If Florence is packed, we should leave," the speaker remains entirely uncertain. This paper, presented at ACL 2019, evaluates how modern NLP models handle these nuances using a challenging, naturalistic dataset.
The CommitmentBank: Capturing the Gradience of Belief
To move beyond "clean" or constructed datasets, the authors utilized the CommitmentBank, a collection of 1,200 naturally occurring items from newswire (Wall Street Journal), fiction (British National Corpus), and spoken dialog (Switchboard). Unlike previous benchmarks that were often biased toward factual examples, the CommitmentBank captures the full spectrum of human belief.
As shown in the figure above, human commitment is highly sensitive to the specific verb used. Factive verbs like know, realize, and notice (highlighted in blue) typically elicit high commitment, while verbs like pretend or seem sit at the opposite end of the spectrum. Crucially, human ratings are gradient; they occupy the entire range from +3 (certainly true) to -3 (certainly false), presenting a significant challenge for models that tend to predict binary outcomes.
Evaluating Rule-Based vs. Neural Models
The researchers compared two distinct paradigms:
- Rule-based Model: A deterministic algorithm that uses "linguistic signatures." It calculates commitment by analyzing specific predicates and their syntactic environments (e.g., the presence of negation).
- Neural Models: Bidirectional Long Short-Term Memory (biLSTM) networks (linear, tree-based, and hybrid). These were trained via multi-task learning across several factuality datasets to learn patterns directly from data.
Performance was measured using Pearson’s $r$ to capture variability and Mean Absolute Error (MAE) to assess prediction accuracy.
Key Results: The Linguistic Advantage
The evaluation revealed a surprising result: the linguistically-informed rule-based model consistently outperformed the more complex neural architectures on the full dataset.
On the full CommitmentBank, the rule-based model achieved a Pearson $r$ of 0.41, nearly double that of the best neural hybrid model (0.23). This suggests that explicit linguistic rules—such as how negation flips the truth value of a complement—are currently more robust than the patterns learned by neural models from limited, often biased, training data.
Identifying the "Linguistic Deficits"
The study highlighted several areas where current deep learning approaches struggle:
- Environment Sensitivity: While models handle simple negation relatively well, they fail significantly on conditionals ("If he believes...") and modals ("He might believe...").
- The "Zero" Point: Models struggle to identify uncertainty. While humans frequently use the middle of the scale to indicate "no commitment," models tend to cluster their predictions at the extremes.
- Genre and Neg-raising: Models performed best on spoken dialog, likely due to the high frequency of simple negation. However, they remain unable to handle "neg-raising" (e.g., "I don't think $p$" meaning "I think not $p$").
Conclusion
The findings of this paper suggest that current state-of-the-art neural models are "brittle" when faced with the diverse syntactic constructions of natural discourse. For AI to achieve a truly robust understanding of speaker commitment, future architectures may need to move beyond raw data and find ways to incorporate deeper, explicit linguistic foreknowledge.
Summary updated