MMG at SemEval-2022 Task 1: A Reverse Dictionary approach based on a review of the dataset from a lexicographic perspective

Alfonso Ardoiz, Miguel Ortega-Martín, Óscar García-Sierra, Jorge Álvarez, Ignacio Arranz, Adrián AlonsoNAACL 2022

TL;DR

This paper tackles the Reverse Dictionary (RD) task, aiming to generate a word from its definition. The authors highlight critical issues with the SemEval-2022 RD dataset's lexicographic quality, proposing preprocessing steps and a novel approach based on lexical graphs and Transformer models to improve performance. Their work underscores the importance of high-quality data for robust NLP tasks.

What Problem Does This Paper Address?

The Reverse Dictionary (RD) task, a component of SemEval-2022 Task 1 (CODWOE), challenges models to generate a word given its natural language definition (gloss). This task is motivated by the need for explainable AI, bridging human-readable definitions with machine-readable word embeddings. However, the effectiveness of RD models is heavily reliant on the quality of the training data. The authors identify a significant problem: the SemEval-2022 RD dataset, while functional, suffers from lexicographical shortcomings. These include a lack of global coherence, inconsistent adherence to definitional principles (like specifying hyperonyms and distinctions), and the inclusion of noise from encyclopedic or overly specific definitions. This poor data quality hinders model convergence and leads to inaccurate results, making it difficult to achieve the precision expected from prestigious dictionaries.

Methods

The MMG team explored several approaches for the RD task. Initially, they focused on leveraging Sentence-Transformers for initializing definition embeddings. A key innovation was the development of a lexical graph constructed from the training data, where words within definitions are linked. This graph was then processed using a SAGE Graph Neural Network (GNN) to capture semantic relationships. For the final model architecture, they employed a Transformer encoder followed by a Multi-Layer Perceptron (MLP). They experimented with two loss functions: Cosine Embedding Loss and Mean Square Error (MSE) Loss, analyzing their trade-offs. Crucially, they also implemented data preprocessing techniques, including removing stopwords and category words, and lemmatizing words in the lexical graph approach, finding preprocessing to be a significant factor in improving performance.

Datasets

The primary dataset used is the SemEval-2022 Task 1 (CODWOE) dataset, specifically the Spanish portion. This dataset is stored in a JSON format and contains "ID," "gloss" (definition), and various embeddings (character-based, Word2Vec Skip-gram Negative Sampling, and ELECTRA where available). The authors note that for the Spanish RD model, they used "sgns" embeddings due to the unavailability of "ELECTRA" embeddings and the perceived limitations of static "char" embeddings.

Key Insights and Ideas

  • Data Quality is Paramount: The paper strongly emphasizes that the quality of lexicographical data is a critical bottleneck for the RD task. Even sophisticated models struggle when trained on noisy or inconsistent definitions.
  • Lexical Graphs Offer Rich Semantic Information: The exploration of lexical graphs, combined with GNNs, demonstrates the potential of explicitly modeling relationships between words within a definition to capture deeper semantic meaning.
  • Loss Function Trade-offs: The study highlights a fundamental tension between optimizing for cosine similarity (semantic closeness) and MSE (direct embedding prediction). Cosine Embedding Loss often yields better semantic similarity but at the cost of higher MSE, suggesting that different metrics might be better suited for different aspects of the task.
  • Preprocessing is Crucial: Simple preprocessing steps like removing stopwords and category words, and lemmatization, significantly improve model performance, underscoring the impact of data cleaning on NLP tasks.

Key Results

The MMG team's model, trained with Cosine Embedding Loss, achieved a cosine similarity score of over 0.4 and a cosine ranking of 0.16 for the Spanish "sgns" embeddings. While this cosine score was competitive, the MSE score was significantly higher than the baseline. The MSE Loss trained model offered a slight improvement in MSE but a worse cosine score. The authors note that their system architecture proved scalable to other languages and embedding types, consistently facing the same trade-offs between cosine similarity and MSE. They also mention that their lexical graph approach, though not submitted due to speed issues, achieved comparable results.

Limitations and Future Work

The authors acknowledge that their results are not valid for the official challenge due to the use of external resources (specifically, the Sentence-Transformers model for initialization). A significant limitation identified is the inherent quality of the SemEval-2022 RD dataset, which they believe could be substantially improved by incorporating more rigorous lexicographical principles. Future work suggested includes further integrating linguistic knowledge, exploring linguistic graphs more deeply, and applying the lessons learned to build better Definition Modeling systems. They also express interest in using more representative contextual embeddings, like "ELECTRA," if they were available for Spanish.

Summary updated

BibTeX
@inproceedings{ardoiz2022mmg,
  author = {Alfonso Ardoiz and Miguel Ortega-Martín and Óscar García-Sierra and Jorge Álvarez and Ignacio Arranz and Adrián Alonso},
  title = {{MMG at SemEval-2022 Task 1: A Reverse Dictionary approach based on a review of the dataset from a lexicographic perspective}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure