daminglu123 at SemEval-2022 Task 2: Using BERT and LSTM to Do Text Classification

Daming LuNAACL 2022

TL;DR

This paper tackles the challenge of identifying idiomatic multiword expressions (MWEs) in text, a task where standard pre-trained language models like BERT struggle due to their reliance on compositional word meanings. The authors propose an enhanced mBERT model by adding an LSTM layer, finding that this simple addition significantly improves performance on this specific classification task.

What Problem Does This Paper Address?

Natural language is rich with multiword expressions (MWEs) – phrases where the combined meaning isn't simply the sum of individual word meanings (e.g., "kick the bucket"). These idiomatic expressions pose a significant challenge for current pre-trained language models (PLMs) like BERT. PLMs are trained to understand language based on the compositional meaning of words, meaning they excel when words combine in predictable ways. However, they often fail to grasp the nuanced, non-literal meanings of idioms. This paper addresses the specific problem of classifying whether a given MWE in a sentence has a trivial (compositional) meaning or an idiomatic one, a task crucial for deeper language understanding.

Methods

The core of the proposed method is building upon the mBERT (multilingual BERT) model. The authors hypothesize that mBERT, being trained on multiple languages, might be better suited for tasks involving diverse linguistic phenomena. To enhance mBERT's ability to capture the contextual nuances of MWEs, they experiment with two common neural network architectures:

  • LSTM (Long Short-Term Memory): LSTMs are known for their ability to remember important information over long sequences, making them suitable for capturing dependencies within sentences. The authors integrate a bidirectional LSTM layer after mBERT's output. This allows the model to consider context from both past and future words when processing the MWE.
  • TextCNN (Text Convolutional Neural Network): Inspired by CNNs in computer vision, TextCNN extracts local features from text. The authors suggest this could help in identifying the specific span or boundaries of MWEs.

They explore adding these layers individually and in combination to mBERT, aiming to improve its performance on the idiomaticity classification task.

Datasets

The paper uses the dataset provided for SemEval-2022 Task 2, Subtask A. This dataset consists of naturally occurring sentences, each with a target MWE. For each MWE, the dataset includes the preceding and succeeding sentences to provide broader context. The task is to classify the MWE as having a compositional (trivial) meaning or an idiomatic meaning. The paper notes that the dataset provides a fine-grained set of meanings, distinguishing between compositional and idiomatic ones.

Key Insights and Ideas

  • Idioms are a blind spot for standard PLMs: The paper highlights that even powerful models like BERT struggle with idiomaticity because their core mechanism relies on compositional word meanings, which doesn't directly translate to understanding non-literal language.
  • Simple architectural additions can help: The surprising insight is that a relatively simple addition of an LSTM layer to an already powerful mBERT model can yield significant improvements for this specific task, suggesting that specialized architectures can still complement large PLMs.
  • Bidirectional context is key: The success of the bidirectional LSTM suggests that understanding the full context surrounding an MWE, both before and after, is crucial for correctly identifying its idiomatic nature.

Key Results

The experiments were conducted on the SemEval-2022 Task 2, Subtask A dataset. The authors compared several configurations:

  • mBERT baseline: Achieved a Zero-Shot score of 0.6448 and a One-Shot score of 0.6987.
  • mBERT + LSTM: With a dropout of 0.1, this configuration achieved the best performance, reaching a Zero-Shot score of 0.6546 and a One-Shot score of 0.6998. When tested with a specific dropout configuration (dp=0.1 test), it further improved to 0.654 and 0.704 respectively.
  • mBERT + TextCNN: This configuration did not perform as well as the LSTM addition, with scores ranging from 0.6501 (Zero-Shot) and 0.6827 (One-Shot) down to 0.6254 (Zero-Shot) and 0.6309 (One-Shot) depending on the dropout rate.
  • mBERT + TextCNN + LSTM: This combined approach also did not outperform the LSTM-only addition.

The paper concludes that mBERT + LSTM performed best for Subtask A among their experiments.

Limitations and Future Work

  • Limited impact of late-stage additions: The authors acknowledge that adding LSTM or TextCNN at the end of the mBERT pipeline might limit their overall impact. They suggest that integrating these components earlier or using different methods, such as framing the problem as a masked word prediction task (as in PROMPT), could lead to more genuine integration and potentially better results.
  • Overfitting concerns: The paper mentions trying to combat overfitting by increasing dropout rates from 0.1 to 0.2, but this had an opposite effect, leading to worse performance. They speculate this might be because their task is text classification, not question answering, where LSTMs are known to boost performance.
  • Future work: While not explicitly detailed, the discussion implies that exploring more integrated approaches to combine PLMs with specialized architectures for MWE understanding would be a promising direction. The authors also provide their code on GitHub, facilitating further research.

Summary updated

BibTeX
@inproceedings{lu2022daminglu,
  author = {Daming Lu},
  title = {{daminglu123 at SemEval-2022 Task 2: Using BERT and LSTM to Do Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure