TL;DR
Large Language Models (LLMs) are getting remarkably good at understanding language, but they still struggle with the nuances of figurative speech like idioms. This paper introduces a new, expert-curated dataset called IdioTS to specifically test LLMs' ability to distinguish between literal and idiomatic meanings of phrases. The study reveals that even advanced conversational LLMs often misinterpret idioms, frequently defaulting to an idiomatic interpretation even when the sentence is literal.
What Problem Does This Paper Address?
Idiomatic expressions are a fundamental part of human language, adding richness and nuance. However, their meanings are often non-literal, making them a significant challenge for Natural Language Processing (NLP) systems, especially Large Language Models (LLMs). While LLMs have shown impressive capabilities in many language tasks, their ability to accurately detect and understand idioms remains an open question. Existing datasets for evaluating figurative language processing are often not specifically designed to test the subtle distinctions LLMs might struggle with, potentially leading to data contamination or insufficient coverage of challenging cases. This paper aims to bridge this gap by creating a robust evaluation benchmark and assessing how well current conversational LLMs perform on this task.
Methods
The core of this research involves creating a new dataset, the Idiomatic language Test Suite (IdioTS), and then using it to evaluate three conversational LLMs: Llama-2-7b-chat, Mistral-7b-Instruct, and Vicuna-7b.
The IdioTS dataset was meticulously constructed by language experts. It includes a curated list of 93 idioms, chosen for their diverse syntactic structures and colloquial style. For each idiom, the researchers crafted new sentences to avoid data contamination. Crucially, the dataset contains both idiomatic (positive class) and distractor (negative class) sentences. Distractor sentences are designed to be plausible and grammatically correct but use words that might appear in an idiom in a literal way, posing a significant challenge. The dataset comprises 164 idiomatic sentences and 86 distractor sentences, totaling 250.
The LLMs were evaluated using an idiom detection task, framed as a binary classification problem: is the sentence idiomatic or not? They were prompted to act as professional linguists and provide their answers in a strict JSON format, including whether an idiom was present, the idiom itself, its meaning, and a concise explanation. The experiment was conducted in a zero-shot setting, meaning no examples were provided in the prompt, to test the models' inherent capabilities.
Datasets
- IdioTS (Idiomatic language Test Suite): A newly introduced dataset specifically designed for evaluating LLMs on idiom detection. It contains 250 sentences, with 164 being idiomatic and 86 being distractor (literal) sentences. The dataset was manually curated by language experts to ensure high quality and cover challenging cases.
Key Insights and Ideas
- LLMs struggle with literal vs. idiomatic distinction: Despite advancements, LLMs still find it difficult to accurately differentiate between sentences where an expression is used idiomatically and those where it's used literally.
- Acquiescence bias is a significant issue: LLMs tend to exhibit "agreement bias," often defaulting to a positive answer (identifying an idiom) even when it's not present. This can lead to them forcing an idiomatic interpretation onto literal sentences.
- Coherent explanations can mask errors: LLMs can generate plausible and coherent explanations for their (incorrect) idiom identifications, which could mislead users who lack the linguistic expertise to verify the output.
- True Positive Consistency is crucial: Simply detecting an idiom isn't enough; the model must also identify the correct idiom. The study found that models sometimes identified a sentence as idiomatic but pointed to the wrong phrase or an unrelated idiom.
- Distractor sentences are effective challenges: The carefully crafted distractor sentences proved effective in highlighting LLMs' weaknesses, particularly in identifying literal uses of potentially idiomatic phrases.
Key Results
- Overall Accuracy: The three LLMs evaluated (Llama2, Mistral, Vicuna) showed similar overall accuracy, ranging from 0.656 to 0.676.
- Mistral excels in Specificity: Mistral demonstrated the best performance in identifying distractor (literal) sentences, with a specificity score of 0.209, significantly outperforming Llama2 (0.0) and Vicuna (0.081). Llama2's 0.0 specificity indicates it incorrectly classified all distractor sentences as idiomatic.
- True Positive Consistency: Mistral also achieved the highest True Positive Consistency (0.905), meaning it was most likely to correctly identify the specific idiom when it classified a sentence as idiomatic. Vicuna followed at 0.889, and Llama2 at 0.841.
- False Positives Dominate: The most common error type across all models was false positives, where idiomaticity was incorrectly attributed to a sentence. False negatives (idiomatic sentences misclassified as literal) were rare, especially for Llama2.
Limitations and Future Work
- Limited Scope: The current study focused on three small-sized, instruction-tuned LLMs and a specific idiom detection task.
- Researcher Bias: The prompt's persona ("professional linguist") and the inclusion of distractor sentences designed to "trap" models could introduce some researcher bias.
- Future Directions: The authors propose extending experiments to one-shot and few-shot scenarios, exploring different prompting techniques, and investigating multi-turn conversations. They also plan to augment the IdioTS dataset, categorize distractor types, and translate the dataset into multiple languages for cross-lingual experiments and machine translation research. Further exploration into how LLM architectures and sizes influence idiom detection is also suggested.
Summary updated