Abstract
Idiomatic expressions present a significant challenge for Natural Language Processing (NLP) due to their non-compositional nature. Standard models often struggle with literal bias and context blindness when tasked with idiom-related cloze tests. This paper introduces IBERT, a novel neural architecture that leverages a dual-attention mechanism combining XLNet and RoBERTa. By processing idioms both as character sequences and as single tokens, IBERT captures both local grammatical fit and global semantic coherence. Experimental results on the EPIE dataset demonstrate that IBERT outperforms previous state-of-the-art models by over 7%, establishing a new benchmark for idiomatic reading comprehension.
Introduction
Idioms are linguistic constructs where the overall meaning cannot be derived from the literal definitions of the individual words. For example, "piece of cake" refers to ease rather than a dessert. Standard NLP models often fail in "cloze-style" tasks—where a missing phrase must be filled in—due to two primary issues:
- Literal Bias: Models frequently interpret figurative expressions literally, failing to associate them with non-literal contexts.
- Context Blindness: Models often over-rely on local context (neighboring words) while ignoring the global narrative context necessary to justify a specific idiom.
Methodology
IBERT addresses these challenges through a "Dual Pretrain Attention" approach. The architecture integrates two distinct perspectives on idiomatic representation to ensure the chosen idiom is both grammatically sound and semantically coherent.
Dual-Path Processing
The model processes idiom candidates through two parallel paths:
- Sequence-based Path (RoBERTa): Treats the idiom as a sequence of tokens. This path uses RoBERTa to embed the passage, focusing on the internal linguistic structure and local relationships between words.
- Token-based Path (XLNet): Treats each idiom as a unique, atomic token. This path utilizes XLNet to capture long-range dependencies and the global "vibe" of the passage.
Linear Interpolation and Pooling
A learnable parameter ($\lambda$) is used to perform linear interpolation between these two perspectives. Furthermore, IBERT employs context-aware pooling, matching idiom candidates against the hidden states of the entire passage rather than just the masked position. This enables the model to detect subtle thematic cues distributed throughout the text.
Experimental Results
The model was evaluated using the EPIE (English Phrases and Idiomatic Expressions) Static Corpus, which contains 21,890 sentences covering 359 idiom types. The dataset was partitioned into training (15k), validation (5k), and test (2k) sets.
IBERT achieved a significant performance improvement over existing baselines. As shown in the table below, it reached an accuracy of 78.23%, surpassing the previous leading model by 7.21% and a standard entity-tracking approach by nearly 15%.
Conclusion and Future Work
IBERT represents a significant advancement in the machine comprehension of figurative language. By balancing global and local context through a hybrid representation, the model effectively mitigates the literal bias inherent in standard transformers.
Future research will focus on expanding the model's evaluation to multilingual benchmarks such as CHID (Chinese) and LIDIOMS. Additionally, the authors plan to explore more sophisticated attention mechanisms, such as block self-attention or hypergraph structures, to better represent the complex multi-word relationships found in idiomatic language.
Summary updated