Abstract
This paper describes our system for the SemEval-2022 task of matching dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track of the competition, which maps multilingual glosses to reconstructed vector representations. More specifically, models convert the input of sentences to three types of embeddings: SGNS, Char, and Electra. We propose several experiments for applying neural network cells, general multilingual and multitask structures, and language-agnostic tricks to the task. We also provide comparisons over different types of word embeddings and ablation studies to suggest helpful strategies. Our initial transformer-based model achieves relatively low performance. However, trials on different retokenization methodologies indicate improved performance. Our proposed Elmo-based monolingual model achieves the highest outcome, and its multitask, and multilingual varieties show competitive results as well.
Introduction
The "Reverse Dictionary" task is a significant challenge in Natural Language Processing (NLP) that involves mapping a human-readable definition (a "gloss") to a machine-readable word representation (an embedding). While standard dictionaries allow users to look up words to find definitions, a reverse dictionary takes a description—such as "a celestial body that orbits a star"—and returns the corresponding word, "planet."
This task requires mapping variable-length sentences into fixed-size vector spaces, specifically Skip-Gram Negative Sampling (SGNS), character-level embeddings (Char), and Electra. The primary difficulties include data scarcity in dictionary datasets compared to massive web corpora, the need for high precision in dense definitions, and the requirement to handle multiple languages (English, Spanish, French, Italian, and Russian).
Methods
The research investigated various architectures and strategies to optimize performance on the SemEval-2022 Task 1 dataset.
Model Architectures and Multitask Learning
The team compared traditional RNNs, LSTMs, Bidirectional RNNs, Elmo, and Transformers. A key innovation was the use of Dynamic Weight Averaging (DWA) for multitask learning. Instead of training a model to predict a single embedding type, the DWA approach allowed the model to predict SGNS, Char, and Electra embeddings simultaneously, facilitating the learning of shared semantic features across different representation styles.
Tokenization and Multilingual Strategies
Three subword tokenization methods were evaluated: Byte Pair Encoding (BPE), Unigram Language Model (ULM), and WordPiece. For multilingual handling, the researchers compared a "language-agnostic" approach (Residual Connection Cutting) against a "language-specific" approach using explicit Language Tags (e.g., [EN], [FR]) prepended to the input.
Key Insights
- The Transformer Trap: Transformer-based models performed poorly due to the relatively small size of the dictionary datasets. Simpler, bidirectional models like Elmo proved more data-efficient.
- Bidirectionality is Essential: Models that processed definitions in both directions (Bi-RNN, Elmo) significantly outperformed unidirectional ones, as crucial context often appears at the end of a definition.
- Language Identity Matters: Explicit language tags outperformed language-agnostic methods, suggesting that maintaining language identity is beneficial for this task.
- Multitask Synergy: Predicting three different embedding types concurrently yielded better results than individual training, indicating that different embedding algorithms capture complementary semantic information.
Key Results
The Elmo + WordPiece + DWA model emerged as the top performer across most metrics. Reconstructing Character and Electra embeddings proved easier for the models than reconstructing SGNS (Word2Vec) embeddings.
Furthermore, a strong correlation was observed between dictionary size and performance; for instance, the Russian dataset, being the largest in the study, yielded the highest cosine similarity scores.
Conclusion
The study demonstrates that for specialized tasks like reverse dictionaries, architectural complexity must be balanced with data availability. While Transformers are powerful, Elmo-based models with multitask learning and explicit language tagging provided superior performance on high-quality, smaller-scale dictionary data. Future work will explore data augmentation techniques to improve Transformer viability and address the inherent complexity of SGNS embeddings.
Summary updated