A Geometrical Approach to Finding Difficult Examples in Language

Debajyoti Datta, Shashwat Kumar, Laura E. Barnes, P. Thomas FletcherTAG-ML 2022

Abstract

State-of-the-art Natural Language Processing (NLP) models often rely on superficial "shortcuts" rather than deep semantic understanding, leading to fragility under minor perturbations. This paper introduces a mathematically grounded method to identify these vulnerabilities using Information Geometry. By analyzing the Fisher Information Metric (FIM) and its largest eigenvalue ($\lambda_{max}$), we quantify the local sensitivity of a model's output to its input. We demonstrate that high $\lambda_{max}$ values correlate strongly with model failure on benchmark tasks, providing a principled way to detect fragile examples without the need for expensive manual curation.

Introduction

Despite achieving human-level performance on standard benchmarks, models like BERT and LSTMs are frequently "right for the wrong reasons." They often engage in shortcut learning, where the model exploits superficial cues—such as specific keywords or sentence structures—to make predictions. Consequently, these models can be highly fragile: a minor, semantically irrelevant change, such as swapping a synonym, can flip the model's prediction.

Existing methods for finding these "difficult" examples often rely on manual human effort or heuristic-based word swapping, which are either unscalable or unprincipled. This research addresses the problem by treating the space of a model's output probability distributions as a non-linear statistical manifold, allowing for a rigorous geometric analysis of model sensitivity.

Methodology: Information Geometry and the Fisher Metric

The core of this approach lies in Information Geometry. Instead of viewing the input space as a flat Euclidean plane, the researchers treat the model's predictions as points on a manifold. The distance between these points is measured using the Fisher Information Metric (FIM), which is essentially the Hessian of the KL divergence between output distributions.

Figure A: Mapping the input space $X$ to the statistical manifold $Z$. The local distortion (stretching of circles into ellipses) represents the Fisher Information Metric, where greater distortion indicates higher sensitivity.

To quantify fragility, the authors focus on $\lambda_{max}$, the largest eigenvalue of the FIM. A high $\lambda_{max}$ indicates a region of high "local distortion"—where a tiny nudge in the input embedding space leads to a massive swing in the model's output confidence. Conversely, a low $\lambda_{max}$ indicates a robust region where predictions remain stable despite perturbations.

Experimental Validation

The researchers validated this geometric metric across several standard NLP benchmarks, including sentiment analysis (IMDB, Yelp) and topic classification (AG_NEWS, Sogou News), using architectures ranging from FastText to BERT.

To visualize the relationship between $\lambda_{max}$ and model behavior, they first applied the method to synthetic data. As shown in the figure below, the points with the highest $\lambda_{max}$ values (indicated by red/orange coloring and larger arrows) are concentrated near the decision boundary. This confirms that the FIM accurately captures the geometric difficulty of an example relative to the model's learned classification logic.

Figure B: Visualization of $\lambda_{max}$ on a synthetic mixture of Gaussians. Points near the decision boundary (dashed line) exhibit higher eigenvalues and sensitivity vectors, marking them as fragile.

Key Results and Insights

  • Correlation with Failure: The study found a consistent correlation (r-values between 0.21 and 0.41) between $\lambda_{max}$ and the probability that a random word substitution would flip the model's label.
  • Accuracy Collapse: When testing on the most "fragile" examples (top 10% of $\lambda_{max}$), model accuracy plummeted to as low as 20% under minor perturbations. In contrast, "robust" examples (bottom 10%) maintained nearly 100% accuracy even under heavy word substitutions.
  • BERT’s Hidden Fragility: While BERT is generally more robust than simpler models, the authors identified high-FIM examples where BERT would flip its sentiment prediction based on a single, irrelevant word change, such as changing a character's name.
  • Meaningless Perturbations: Fragile examples are susceptible to "meaningless" changes. For instance, in a positive movie review, changing "favorite" to "preferred" was enough to make a fragile model classify the review as negative.

Conclusion and Future Work

This geometric approach provides a principled, architecture-agnostic framework for identifying the "cracks" in NLP models. By focusing on the largest eigenvalue of the Fisher Information Metric, researchers can pinpoint exactly which inputs a model is most uncertain about, even when its confidence scores appear high.

Future work will address the "decoding" problem: while the method identifies the direction in embedding space that causes the most fragility, translating that abstract vector back into a natural-sounding, human-readable sentence remains a challenge. Additionally, extending this framework to generative tasks like translation or summarization could provide deeper insights into the stability of modern language generation.

Summary updated

BibTeX
@inproceedings{datta2022geometrical,
  author = {Debajyoti Datta and Shashwat Kumar and Laura E. Barnes and P. Thomas Fletcher},
  title = {{A Geometrical Approach to Finding Difficult Examples in Language}},
  booktitle = {TAG-ML},
  year = {2022},
}
Figure