IBERT: Idiom Cloze-style reading comprehension with Attention

Ruiyang Qin, Haozheng Luo, Zheheng Fan, Ziang RenarXiv 2021 · arXiv

Abstract

Idiomatic expressions present a significant challenge for Natural Language Processing (NLP) due to their non-compositional nature. Standard models often struggle with literal bias and context blindness when tasked with idiom-related cloze tests. This paper introduces IBERT, a novel neural architecture that leverages a dual-attention mechanism combining XLNet and RoBERTa. By processing idioms both as character sequences and as single tokens, IBERT captures both local grammatical fit and global semantic coherence. Experimental results on the EPIE dataset demonstrate that IBERT outperforms previous state-of-the-art models by over 7%, establishing a new benchmark for idiomatic reading comprehension.

Introduction

Idioms are linguistic constructs where the overall meaning cannot be derived from the literal definitions of the individual words. For example, "piece of cake" refers to ease rather than a dessert. Standard NLP models often fail in "cloze-style" tasks—where a missing phrase must be filled in—due to two primary issues:

  1. Literal Bias: Models frequently interpret figurative expressions literally, failing to associate them with non-literal contexts.
  2. Context Blindness: Models often over-rely on local context (neighboring words) while ignoring the global narrative context necessary to justify a specific idiom.

Methodology

IBERT addresses these challenges through a "Dual Pretrain Attention" approach. The architecture integrates two distinct perspectives on idiomatic representation to ensure the chosen idiom is both grammatically sound and semantically coherent.

The IBERT architecture utilizes a dual-path approach combining XLNet and RoBERTa.

Dual-Path Processing

The model processes idiom candidates through two parallel paths:

  • Sequence-based Path (RoBERTa): Treats the idiom as a sequence of tokens. This path uses RoBERTa to embed the passage, focusing on the internal linguistic structure and local relationships between words.
  • Token-based Path (XLNet): Treats each idiom as a unique, atomic token. This path utilizes XLNet to capture long-range dependencies and the global "vibe" of the passage.

Linear Interpolation and Pooling

A learnable parameter ($\lambda$) is used to perform linear interpolation between these two perspectives. Furthermore, IBERT employs context-aware pooling, matching idiom candidates against the hidden states of the entire passage rather than just the masked position. This enables the model to detect subtle thematic cues distributed throughout the text.

Experimental Results

The model was evaluated using the EPIE (English Phrases and Idiomatic Expressions) Static Corpus, which contains 21,890 sentences covering 359 idiom types. The dataset was partitioned into training (15k), validation (5k), and test (2k) sets.

IBERT achieved a significant performance improvement over existing baselines. As shown in the table below, it reached an accuracy of 78.23%, surpassing the previous leading model by 7.21% and a standard entity-tracking approach by nearly 15%.

Performance comparison on the EPIE dataset shows IBERT outperforming existing baselines.

Conclusion and Future Work

IBERT represents a significant advancement in the machine comprehension of figurative language. By balancing global and local context through a hybrid representation, the model effectively mitigates the literal bias inherent in standard transformers.

Future research will focus on expanding the model's evaluation to multilingual benchmarks such as CHID (Chinese) and LIDIOMS. Additionally, the authors plan to explore more sophisticated attention mechanisms, such as block self-attention or hypergraph structures, to better represent the complex multi-word relationships found in idiomatic language.

Summary updated

Abstract

Idioms are special fixed phrases usually derived from stories. They are commonly used in casual conversations and literary writings. Their meanings are usually highly non-compositional. The idiom cloze task is a challenge problem in Natural Language Processing (NLP) research problem. Previous approaches to this task are built on sequence-to-sequence (Seq2Seq) models and achieved reasonably well performance on existing datasets. However, they fall short in understanding the highly non-compositional meaning of idiomatic expressions. They also do not consider both the local and global context at the same time. In this paper, we proposed a BERT-based embedding Seq2Seq model that encodes idiomatic expressions and considers them in both global and local context. Our model uses XLNET as the encoder and RoBERTa for choosing the most probable idiom for a given context. Experiments on the EPIE Static Corpus dataset show that our model performs better than existing state-of-the-arts.
BibTeX
@misc{qin2021ibert,
  author = {Ruiyang Qin and Haozheng Luo and Zheheng Fan and Ziang Ren},
  title = {{IBERT: Idiom Cloze-style reading comprehension with Attention}},
  year = {2021},
  eprint = {2112.02994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.02994},
}
Figure