Revisiting a Pain in the Neck: Semantic Phrase Processing Benchmark for Language Models

Yang Liu, Melissa Xiaohui Qin, Hongming Li, Chao Huang

Introduction

In the field of linguistics, Semantic Phrases (SPs)—frequently referred to as Multiword Expressions (MWEs)—have long been described as a "pain in the neck" for Natural Language Processing. These word combinations, such as idioms ("kick the bucket") or collocations ("heavy rain"), possess meanings that are often non-compositional, meaning the total sense is not merely the sum of the individual words. For Large Language Models (LLMs), these phrases serve as a critical litmus test for true linguistic understanding versus sophisticated statistical pattern matching.

Despite the rapid advancement of LLMs, evaluation of their performance on SPs has remained fragmented, often focusing on isolated phenomena like idioms or noun compounds. To address this, researchers have introduced LEXBENCH, the first comprehensive benchmark designed to evaluate how diverse language model architectures handle the full spectrum of semantic phrases across detection, extraction, and interpretation tasks.

The LEXBENCH Framework

LEXBENCH provides a unified framework covering four major linguistic categories:

  1. Idiomatic Expressions (IE): Non-compositional, figurative phrases (e.g., "bite the bullet").
  2. Noun Compounds (NC): Combinations where meanings range from literal to figurative (e.g., "sweet victory").
  3. Verbal Constructions (VC): Semi-compositional verb-based phrases (e.g., "take a decision").
  4. Lexical Collocations (LC): Frequent word pairings (e.g., "strong tea").

The benchmark evaluates 15 different models, ranging from encoder-only models like BERT to state-of-the-art generative models like GPT-4 and Claude-3-Opus, across 10 distinct tasks. The researchers also introduced Oracle Prompting, a technique that provides models with formal linguistic definitions to assess if explicit expert knowledge improves performance in extraction tasks.

Key Insights: The Interpretation-Extraction Paradox

A primary finding of the study is the "Interpretation-Extraction Paradox." While modern LLMs have become exceptionally good at explaining what a phrase means (Interpretation), they continue to struggle with identifying the exact boundaries of those same phrases within a sentence (Extraction).

Figure 1: Radar charts showing model performance across categorization, interpretation, and extraction for different phrase types.

As shown in Figure 1, the performance across different phrase types is uneven. Models generally excel at interpretation but show a marked decrease in capability when tasked with extraction. This suggests that while LLMs can "talk about" a phrase, they often fail to "see" its structural limits in context.

The study reveals several critical trends regarding model size and task complexity:

  • Small Models vs. LLMs: In specific classification tasks, such as idiom detection, smaller fine-tuned models like BERT-large can outperform zero-shot LLMs, indicating that specialized training remains valuable for niche linguistic tasks.
  • The Decay of Accuracy: Model performance is highly sensitive to the number of semantic categories involved in a task. As the classification space expands, even the most advanced models show a significant decline in accuracy.
Figure 4: Mean accuracy of various models as the number of semantic categories increases.

Figure 4 illustrates this "stochastic parrot" warning: as the number of semantic categories increases from 1 to 16, the mean accuracy of models like ChatGPT and GPT-4 decays significantly. In the most complex 16-category scenarios, zero-shot performance for some models regresses toward a random baseline.

Conclusion

LEXBENCH demonstrates that while LLMs have made significant strides in linguistic fluency, semantic phrase processing remains a formidable challenge, particularly for generative extraction. GPT-4 and Claude-3-Opus currently lead the field, showing the most balanced capabilities. However, the performance gap in extraction and the sensitivity to task complexity suggest that simply increasing parameter counts may not be enough to solve the idiosyncratic nature of semantic phrases. Future work will focus on expanding the benchmark to multilingual contexts and exploring "long-tail" linguistic phenomena to further push the boundaries of model understanding.

Summary updated

Abstract

We introduce LexBench, a comprehensive evaluation suite enabled to test language models (LMs) on ten semantic phrase processing tasks. Unlike prior studies, it is the first work to propose a framework from the comparative perspective to model the general semantic phrase (i.e., lexical collocation) and three fine-grained semantic phrases, including idiomatic expression, noun compound, and verbal construction. Thanks to \ourbenchmark, we assess the performance of 15 LMs across model architectures and parameter scales in classification, extraction, and interpretation tasks. Through the experiments, we first validate the scaling law and find that, as expected, large models excel better than the smaller ones in most tasks. Second, we investigate further through the scaling semantic relation categorization and find that few-shot LMs still lag behind vanilla fine-tuned models in the task. Third, through human evaluation, we find that the performance of strong models is comparable to the human level regarding semantic phrase processing. Our benchmarking findings can serve future research aiming to improve the generic capability of LMs on semantic phrase comprehension. Our source code and data are available at https://github.com/jacklanda/LexBench
BibTeX
@misc{liu2024revisiting,
  author = {Yang Liu and Melissa Xiaohui Qin and Hongming Li and Chao Huang},
  title = {{Revisiting a Pain in the Neck: Semantic Phrase Processing Benchmark for Language Models}},
  year = {2024},
  eprint = {2405.02861},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.02861},
}
Figure