Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers

Shane Storks, Joyce Chai

Abstract

As large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, statistical bias in benchmark data and probing studies have recently called into question their true capabilities. For a more informative evaluation than accuracy on text classification tasks can offer, we propose evaluating systems through a novel measure of prediction coherence. We apply our framework to two existing language understanding benchmarks with different properties to demonstrate its versatility. Our experimental results show that this evaluation framework, although simple in ideas and implementation, is a quick, effective, and versatile measure to provide insight into the coherence of machines’ predictions.

Introduction

High accuracy in Natural Language Processing (NLP) has become so commonplace that "human-level performance" is often met with skepticism. While modern models are undeniably powerful, a critical question remains: are they truly understanding the text, or are they merely sophisticated pattern matchers exploiting statistical shortcuts?

Researchers at the University of Michigan address this in their paper, "Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers." They argue that standard accuracy metrics only capture the "tip" of a model's capability. To understand the underlying reasoning, we must measure prediction coherence—the alignment between a model's internal logic and human reasoning across a discourse.

The 'Iceberg' metaphor for conversational entailment: accuracy on the final prediction is only the tip; coherence requires understanding the underlying evidence.

The "Clever Hans" Problem in NLP

The NLP community currently faces a "Clever Hans" dilemma, where Large Language Models (LLMs) achieve high scores by latching onto spurious correlations or statistical "cheats" in datasets rather than performing robust reasoning. For instance, in conversational entailment, a model might correctly predict a hypothesis is true based on keyword presence, while ignoring the specific sentences that provide the logical proof. For a model to be trustworthy, its reasoning must be coherent across the entire input.

Methodology: Assessing Sub-Spans

The proposed framework evaluates models on sub-spans of the original input through a three-step process:

  1. Decomposition: Input is broken into consecutive sub-spans (turns, sentences, or paragraphs).
  2. Human Ground Truth: Annotators identify which sub-spans contain sufficient evidence to support a conclusion.
  3. Coherence Metrics:
    • Strict Coherence: An all-or-nothing measure where a model must correctly predict the end task and provide correct labels for every sub-span.
  4. Lenient Coherence: An average of the model's accuracy across all sub-spans.
  5. Confidence Thresholding: A coherence parameter ($\rho$) ensures models show low confidence when presented with insufficient information.

Experimental Setup

The framework was tested on two distinct benchmarks:

  • Conversational Entailment (CE): Deciding if a hypothesis is supported by a multi-turn dialogue.
  • Abductive Reasoning in narrative Text (ART): A large-scale commonsense reasoning dataset where models choose the most plausible intermediate event between two given sentences.

Key Results and the Accuracy-Coherence Gap

The results reveal a significant "hidden" failure rate. Models often maintain high accuracy while suffering massive drops in coherence, suggesting they frequently "guess" correctly based on fragments rather than holistic processing.

Conversational Entailment Results

Model (on CE) Accuracy Strict Coherence Drop ($\Delta$)
BERT 55.8% 28.5% -27.3%
RoBERTa 70.9% 39.0% -31.9%
DeBERTa 67.4% 37.2% -30.2%

Abductive Reasoning (ART) Results

On the ART dataset, the performance gap was even more pronounced. While DeBERTa achieved a high accuracy of 88.4%, its strict coherence plummeted to 59.8%. This indicates that in nearly 30% of cases, the model reached the correct conclusion but failed to correctly identify the supporting evidence or showed misplaced confidence in incomplete stories.

Performance comparison on the ART dataset, showing significant drops between standard accuracy and strict/lenient coherence metrics.

Discussion and Conclusion

The study highlights that DeBERTa is generally the most coherent model, likely due to architectural features like disentangled attention. However, the overall coherence gap remains large across all models. Interestingly, transfer learning (e.g., pre-training on MultiNLI) improved accuracy but failed to close the coherence gap, indicating that more data does not inherently teach reasoning.

While the framework requires additional human annotation, it provides a necessary "rationale check" for future AI development. Moving forward, benchmarks should incorporate these intermediate checks to drive the creation of truly human-aligned reasoning systems.

Summary updated

Abstract

As large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, statistical bias in benchmark data and probing studies have recently called into question their true capabilities. For a more informative evaluation than accuracy on text classification tasks can offer, we propose evaluating systems through a novel measure of prediction coherence. We apply our framework to two existing language understanding benchmarks with different properties to demonstrate its versatility. Our experimental results show that this evaluation framework, although simple in ideas and implementation, is a quick, effective, and versatile measure to provide insight into the coherence of machines’ predictions.
BibTeX
@inproceedings{storks2021beyond,
  author = {Shane Storks and Joyce Chai},
  title = {{Beyond the Tip of the Iceberg: Assessing Coherence of Text Classifiers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=aBXKkVg9G2},
}
Figure