RIGA at SemEval-2022 Task 1: Scaling Recurrent Neural Networks for CODWOE Dictionary Modeling

Eduards Mukans, Gus Strazds, Guntis BarzdinsNAACL 2022

TL;DR

This paper explores scaling Recurrent Neural Networks (RNNs) for the challenging task of generating dictionary glosses from word embeddings. The authors demonstrate that scaling RNNs in depth and width, similar to trends seen in Transformers, leads to improved performance, and they observe a "double descent" phenomenon where performance can improve even after initial overfitting.

What Problem Does This Paper Address?

The SemEval-2022 Task 1: CODWOE (Comparing Dictionaries and WOrd Embeddings) challenged participants to generate dictionary glosses from individual word embedding vectors. This is a non-trivial task because it requires translating a dense numerical representation of a word's meaning into a natural language definition. While scaling laws and their impact on performance have been extensively studied for Transformer models, their effect on traditional Recurrent Neural Networks (RNNs) like LSTMs and GRUs has been less explored. This paper aims to fill that gap by investigating how scaling RNNs in terms of model size (depth and width) and training duration affects their performance on this definition modeling task.

Methods

The authors employed classical recurrent neural network architectures, specifically Gated Recurrent Units (GRUs), for the definition modeling task. Their approach involved:

  • Input Layer: A fully connected input layer was used to scale the word embedding vectors to higher dimensions.
  • Decoder-Only Seq2Seq: They utilized a decoder-only part of a standard sequence-to-sequence (seq2seq) model without attention mechanisms.
  • Teacher Forcing: To mitigate error accumulation during training, they employed teacher forcing, where the model is fed ground-truth tokens from the target sequence. They also experimented with scheduled sampling, but found it less effective for this task.
  • Model Scaling: The core of their investigation involved systematically scaling the GRU models in two ways:
  • Depth: Increasing the number of layers.
  • Width: Increasing the hidden dimension size.
  • Training Duration: They also explored the effect of training for extended periods to observe potential "double descent" phenomena.
  • Automatic Metrics: Performance was evaluated using standard metrics like MoverScore, S-BLEU, and L-BLEU, as well as a custom accuracy metric (percentage of correctly predicted tokens assuming previous tokens were correct).

Datasets

The primary dataset used was the CODWOE SemEval-2022 Task 1 dataset, which provides word embedding vectors and corresponding dictionary glosses for five languages: English, French, Spanish, Italian, and Russian. The authors also used a trial dataset consisting of 200 glosses for most of their experiments and a development (validation) dataset for accuracy calculations. No external data was permitted.

Key Insights and Ideas

  • RNNs Benefit from Scaling: Contrary to the notion that RNNs might be less amenable to scaling than Transformers, this paper shows that increasing the depth and width of GRU models significantly improves performance on the definition modeling task.
  • Double Descent Phenomenon in RNNs: The authors observed "epoch-wise double descent," where training for longer periods initially leads to overfitting but then performance can improve again after a certain point. This effect was more pronounced in larger models.
  • Vocabulary Size Matters: They found that an optimal vocabulary size, around 10% of the unique word tokens in the training data, yielded the best results, confirming their hypothesis.
  • MoverScore Limitations: The paper highlights that MoverScore, while correlating with human evaluation, can be quite variable and sometimes produce negative scores, especially when gloss lengths differ significantly from the ground truth. This suggests a need for better automatic evaluation metrics.

Key Results

  • The "emukans" and "guntis" submissions achieved competitive results, placing in the middle of the overall ranking.
  • Specifically, their solution achieved the top score for French, 2nd for Spanish, and 3rd for Russian on the MoverScore metric.
  • Scaling the GRU model in depth (more layers) and width (larger hidden dimensions) consistently led to performance improvements across various automatic metrics. For instance, increasing hidden dimensions from 512 to 3072 for a 2-layer GRU model improved MoverScore.
  • The optimal vocabulary size was found to be around 1500 tokens, which was approximately 8.3% of the unique word tokens, supporting their hypothesis.
  • Manual evaluation indicated that glosses generated later in training (after the initial accuracy spike) were semantically closer to the original words, confirming the double descent effect's positive impact.

Limitations and Future Work

  • Compute Limitations: The authors acknowledge that their experiments were limited by available compute resources, preventing them from scaling models to extremely large sizes. This means they could not fully explore the extent of the double descent effect.
  • Metric Sensitivity: The automatic metrics (MoverScore, BLEU) were found to be less sensitive than a custom accuracy metric or human judgment, particularly in capturing subtle improvements after initial overfitting.
  • Future Work:
    • Further investigation into the double descent effect with more extensive scaling.
  • Developing better automatic metrics that more closely align with human judgment for evaluating generated glosses.
  • Exploring the possibility that a single vector might be insufficient to represent word semantics, suggesting that tasks involving synonyms or semantically related words might be more appropriate.
  • The potential benefit of larger training datasets or the use of external data/pretrained language models is also suggested.

Summary updated

BibTeX
@inproceedings{mukans2022riga,
  author = {Eduards Mukans and Gus Strazds and Guntis Barzdins},
  title = {{RIGA at SemEval-2022 Task 1: Scaling Recurrent Neural Networks for CODWOE Dictionary Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure