Abstract
This paper describes our system for the SemEval2022 task of matching dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track of the competition, which maps multilingual glosses to reconstructed vector representations. More specifically, models convert the input of sentences to three types of embeddings: SGNS, Char, and Electra. We propose several experiments for applying neural network cells, general multilingual and multitask structures, and language-agnostic tricks to the task. We also provide comparisons over different types of word embeddings and ablation studies to suggest helpful strategies. Our initial transformer-based model achieves relatively low performance. However, trials on different retokenization methodologies indicate improved performance. Our proposed Elmo-based monolingual model achieves the highest outcome, and its multitask and multilingual varieties show competitive results as well.
Introduction
The "Reverse Dictionary" task addresses a common linguistic phenomenon: knowing a definition but being unable to recall the specific word. In computational linguistics, this translates to mapping a natural language description (e.g., "a celestial body that orbits a star") to a precise point in a high-dimensional vector space, such as a word embedding for "planet."
Participating in the SemEval-2022 Task 1 (CODWOE), the 1Cademy team faced the challenge of reconstructing three distinct embedding types—SGNS (Word2Vec), Char (character-based), and Electra (transformer-based)—across five languages. A key constraint was the prohibition of massive pre-trained models, requiring systems to be trained from scratch on relatively small dictionary datasets.
Methodology
The proposed system architecture focuses on balancing model complexity with data scarcity. The core of the approach involves a multi-layer bidirectional LSTM structure (Elmo-style) to capture full contextual information from definitions.
Model Components
- Architecture: The team compared standard RNNs, LSTMs, and Transformers against an Elmo-based structure (multi-layer Bi-LSTM).
- Multitask Learning: To predict SGNS, Char, and Electra embeddings simultaneously, the system employs Dynamic Weight Averaging (DWA). This mechanism adjusts the loss contribution of each task during training to ensure balanced convergence.
- Multilingual Strategies: The model handles English, Spanish, French, Italian, and Russian. Two primary strategies were tested: a shared vocabulary and the use of Language Tags (e.g.,
[EN],[FR]) to provide explicit linguistic signaling. - Retokenization: Subword tokenization methods including BPE, WordPiece, and ULM were evaluated for their effectiveness in handling dictionary-specific vocabularies.
Results and Analysis
The experimental results highlight several critical insights regarding model performance in low-resource settings.
Key Findings
- The Transformer Paradox: Contrary to trends in large-scale NLP, Transformer-based models performed poorly in this task. Their high complexity led to overfitting on the small dictionary datasets.
- Superiority of Bidirectionality: Elmo-based models consistently outperformed unidirectional alternatives. Reading definitions "forwards and backwards" proved essential for pinpointing semantic targets.
- Multitask Synergy: Training on all three embedding types simultaneously improved overall results, suggesting that different mathematical representations share a common semantic core.
- Embedding Difficulty: SGNS embeddings were the most difficult to reconstruct. This likely stems from SGNS capturing broad contextual information from massive corpora that is absent in concise dictionary definitions.
The Elmo + WordPiece + DWA configuration emerged as the top performer. In English benchmarks, it achieved a Mean Squared Error (MSE) of 0.985 for SGNS, outperforming the standard Transformer's 1.042. Additionally, language-specific tokens (ALT) provided a consistent performance boost across languages.
Conclusion
The study demonstrates that for specialized, data-constrained tasks like the Reverse Dictionary, optimized Bi-LSTM structures combined with multitask learning can outperform more complex Transformer architectures. Future work will focus on bridging the gap between "pure" dictionary definitions and contextual usage in large corpora, as well as exploring dictionary-based debiasing to create more neutral word representations.
Summary updated