No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models

Kellen Cheng, Suma BhatNAACL 2024 · DOI

Abstract

Idiomatic expressions (IEs) present a significant challenge for Natural Language Processing due to their non-compositional nature. While humans rely on surrounding context to decode figurative meanings, this study reveals that general-purpose pre-trained language models (PTLMs) often exhibit an "inverse context effect," performing better when context is removed. We investigate this "contextual quandary" across various benchmarks, finding a sharp divide between standard models and those specifically engineered for idiomatic awareness.

Introduction: The Challenge of Non-Compositionality

Idioms like "squaring the circle" cannot be understood by simply aggregating the meanings of their constituent words. In human communication, the surrounding sentential and extra-sentential context provides the necessary cues to resolve these figurative meanings.

Figure 1: Conceptual overview of idiomatic reasoning. The top panel shows how context typically aids in mapping an idiom to its figurative meaning, while the bottom panel illustrates the ambiguity or failure that occurs in isolation.

Researchers have long assumed that large language models (LLMs) mirror this human reliance on context. However, this paper questions whether models are truly "reasoning" through context or merely relying on memorized shortcuts that context might inadvertently obscure.

Methodology

The study employs a data-driven approach to probe model behavior by systematically modifying input context:

  1. Context Removal: Stripping away surrounding sentences (extra-sentential) or non-idiom words within the same sentence (sentential).
  2. Context Shuffling: Permuting context words to destroy syntax while preserving the "bag of words" signal.
  3. Random String Replacement: Replacing idioms with nonsense strings (e.g., "abcde") to test if models can infer meaning from context alone.

The evaluation compares general-purpose models (e.g., BART, Mistral) against IE-aware models (e.g., BART-IEKG, PIER+) which incorporate idiomatic knowledge bases. Experiments were conducted on benchmarks including IMPLI (idiomatic NLI), FigurativeNarrativeBenchmark, and FLUTE.

Key Insights: The Contextual Quandary

The experiments revealed several counter-intuitive findings regarding how PTLMs handle figurative language:

  • The Inverse Context Effect: For standard models, context often acts as noise. On the IMPLI dataset, models like BART and Mistral-FT showed performance gains when context was removed, suggesting they rely on shallow heuristics rather than contextual reasoning.
  • The IE-Aware Divide: Models augmented with idiomatic knowledge behave more like humans. Their performance drops significantly when context is removed, indicating they have successfully learned to utilize surrounding cues.
  • Frequency Bias: Models utilize context more effectively for high-frequency idioms. Rare expressions often lead to confusion, where the context interferes with the model's internal representation of the idiom.

Results and Performance Analysis

The performance delta ($\delta$), defined as the difference between "No Context" and "With Context" accuracy, highlights the disparity between model types.

Figure 2: Performance difference ($\delta$) on the IMPLI dataset. Positive values indicate better performance without context (the "Inverse Context Effect"), seen in general models like BART and Mistral-FT. Negative values indicate a reliance on context, characteristic of IE-aware models like PIER+.

As shown in Figure 2, BART-MNLI’s accuracy increased by 3.89% (from 77.87% to 81.76%) when context was stripped. Conversely, the IE-aware PIER+ model suffered a performance drop, particularly on the FigurativeNarrativeBenchmark where accuracy fell from 65.08% to 60.54% upon context removal. In the random string replacement test, IE-aware models demonstrated the highest context utilization, with accuracy drops of up to 17.77% when context was unavailable.

Discussion and Future Work

The findings suggest that general PTLMs do not "reason" about idioms in the human sense; instead, they may be hindered by the very context meant to aid them. This could be due to spurious correlations in training data or failures in attention mechanisms to correctly weight idiomatic tokens.

Future research should investigate the underlying causes of this contextual interference. The authors advocate for the development of more robust, IE-aware training paradigms to bridge the gap between model heuristics and human-like linguistic reasoning.

Summary updated

BibTeX
@inproceedings{cheng2024no,
  author = {Kellen Cheng and Suma Bhat},
  title = {{No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.272},
}
Figure