Towards a Transformer-Based Reverse Dictionary Model for Quality Estimation of Definitions (Student Abstract)

Julien Guité-Vinet, Alexandre Blondin Massé, Fatiha SadatAAAI 2024 · DOI

Introduction

In traditional lexicography, a dictionary serves to map a known word to its unknown meaning. The "reverse dictionary" task flips this paradigm: given a descriptive definition (e.g., "a celestial body moving in an elliptical orbit round a star"), the system must retrieve the corresponding target word ("planet"). This task represents a significant challenge in Natural Language Processing (NLP) due to the complexities of synonymy, polysemy, and the varied ways humans conceptualize semantic space.

This research explores the efficacy of transformer-based models in solving the reverse dictionary problem. Beyond the utility of helping users find words on the "tip of their tongue," this work aims to establish a framework for evaluating the quality of human-written definitions. Specifically, the authors apply these models to data from The Dictionary Game, a serious game designed to map the human mental lexicon. By measuring how easily a model can identify a word from a player's description, the researchers can objectively estimate the quality and clarity of that definition.

Methodology

The study evaluates three distinct transformer architectures to determine which best captures the nuances of definitional text:

  • DistilBERT: A compressed version of BERT that utilizes a bidirectional encoder. It processes the entire definition simultaneously, allowing it to capture context from both the beginning and end of a sentence.
  • DistilGPT-2: A distilled version of GPT-2. As an autoregressive decoder, it processes text linearly from left to right, predicting subsequent tokens based on preceding ones.
  • XLNet: A model that integrates features of both architectures. It uses a permutation-based training method to capture bidirectional context while maintaining an autoregressive formulation.

Experimental Setup

The models were fine-tuned with an additional dropout layer and a softmax layer for classification. To improve performance, the researchers introduced an auxiliary task: the models were required to predict the Part of Speech (POS) of the target word. This additional signal helps the model narrow the potential search space within the vocabulary. For computational efficiency, input definitions were capped at 50 tokens.

Results and Performance

The models were benchmarked using the standard dataset from Hill et al. (2016). Performance was measured using the median rank of the target word (where lower is better) and Top-N accuracy (the frequency with which the correct word appears in the top 1, 10, or 100 results).

Model Comparison

The results demonstrated that bidirectionality is essential for the reverse dictionary task. XLNet and DistilBERT significantly outperformed the unidirectional DistilGPT-2. This suggests that understanding a definition requires the model to synthesize the entire semantic string at once, rather than processing it as a sequence of independent predictions.

XLNet emerged as the superior model, achieving:

  • Median Rank: 3
  • Top-1 Accuracy: 0.38
  • Top-10 Accuracy: 0.65
  • Top-100 Accuracy: 0.85

These figures represent a notable improvement over previous state-of-the-art benchmarks. Interestingly, while DistilBERT showed higher "certainty" in its predictions (a probability score of 0.55), it frequently struggled with synonyms. The researchers concluded that high probability scores can be misleading; a model might be highly certain about a semantically related synonym while failing to identify the exact target word required by the dataset.

Estimating Definition Quality

A core contribution of this work is the proposal that "average rank" serves as a robust metric for definition quality. In analyzing The Dictionary Game data, the researchers found that smaller dictionaries (those with fewer than 24 words) often yielded better average ranks than larger ones. This suggests that a more constrained, interconnected vocabulary may lead to more precise and effective definitions.

Limitations and Future Directions

The current study is limited by the 50-token constraint on definitions and the inherent "synonym problem," where models are penalized for correct semantic guesses that do not match the specific gold-standard label.

Future work will focus on:

  1. Expanding Data Diversity: Incorporating non-traditional sources like Urban Dictionary for slang and crossword puzzle clues for lateral thinking challenges.
  2. Refining Metrics: Developing more sophisticated qualitative measures to provide feedback to players of The Dictionary Game, helping them understand how to construct more effective and retrievable definitions.

Summary updated

BibTeX
@inproceedings{guitevinet2024towards,
  author = {Julien Guité-Vinet and Alexandre Blondin Massé and Fatiha Sadat},
  title = {{Towards a Transformer-Based Reverse Dictionary Model for Quality Estimation of Definitions (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30449},
}
Figure