Sõnajaht: Definition Embeddings and Semantic Search for Reverse Dictionary Creation

Aleksei Dorkin, Kairit SirtsNAACL 2024

TL;DR

This paper introduces "Sõnajaht," an information retrieval-based reverse dictionary system that leverages modern pre-trained language models and approximate nearest neighbor search. It enhances an existing Estonian lexicon by enabling cross-lingual semantic search, allowing users to find words by describing their meaning in their native language. The work also proposes a novel, unlabeled evaluation method that uses synonymy relations within the lexicon itself, reducing reliance on manually annotated data.

What Problem Does This Paper Address?

Reverse dictionaries are valuable tools for writers, language learners, and anyone experiencing a "tip-of-the-tongue" moment. They allow users to input a definition or description and retrieve the corresponding word or phrase. Traditional approaches often rely on keyword matching, which struggles with the nuances of language. More recent methods use machine learning to reconstruct word embeddings or fine-tune transformer models, but these typically require extensive labeled datasets, limiting their applicability to new languages or domains. This paper addresses the challenge of building effective reverse dictionaries, particularly for languages like Estonian, where labeled data is scarce, and aims to enable cross-lingual functionality without extensive manual annotation.

Methods

The core of Sõnajaht is an information retrieval (IR) approach that uses pre-trained language models to create dense vector representations of word definitions. These definitions form a search index stored in a vector database. When a user provides a description, it's also encoded into a vector, and an approximate nearest neighbor (ANN) search algorithm (specifically HNSW) is used to find the most similar definition vectors. This approach is novel in its application to reverse dictionaries, combining established techniques for semantic search. The system is built upon the existing Estonian lexicon resource, Sõnaveeb, and its code and data are made publicly available.

Datasets

  • Sõnaveeb: The primary Estonian language lexicon used as the source for words, definitions, and synonymy relations. This resource forms the basis of the system and the unlabeled evaluation dataset.
  • Extended English Dataset: An existing labeled dataset of English words and definitions (from Hill et al., 2016) was extended to include Estonian and Russian translations for cross-lingual evaluation.

Key Insights and Ideas

  • IR-based approach is feasible without extensive training: The paper demonstrates that a well-performing reverse dictionary can be built using pre-trained models and IR techniques without task-specific model training, which is a significant advantage for low-resource languages.
  • Novel unlabeled evaluation is effective: The proposed method of using synonymy relations within the lexicon to create ground truth for evaluation is a clever way to bypass the need for manually annotated data, making evaluation more accessible.
  • Cross-lingual retrieval models excel: Models specifically trained for cross-lingual retrieval and that include the target language (Estonian) in their training data consistently outperform others, highlighting the importance of language coverage and retrieval-specific training.
  • Simple Word2Vec can be surprisingly competitive: The Estonian Word2Vec baseline, despite its simplicity, performed comparably to or even better than some larger multilingual models like XLM-ROBERTa in certain settings, suggesting that even simpler embeddings can be effective for specific tasks.

Key Results

  • Monolingual performance: The system achieved a median rank of 1 in the monolingual setting using the unlabeled evaluation approach.
  • Cross-lingual performance: In the cross-lingual setting, the median rank was 2.
  • Top models: E5 Embeddings (with query-passage encoding) and LaBSE consistently ranked highest across various metrics (MAP, MP@1, MRR, Acc@1, Acc@10) in both unlabeled and labeled evaluations.
  • OpenAI's model underperforms: The proprietary OpenAI model showed relatively poor performance, especially in cross-lingual settings, suggesting its training might not be optimized for this specific task.
  • XLM-ROBERTa's surprising performance: Despite being a powerful multilingual model, XLM-ROBERTa performed less well than expected, even worse than the simple Word2Vec baseline in some cases.

Limitations and Future Work

  • Lack of human evaluation: The authors acknowledge that while automatic metrics are used, direct human judgment on user satisfaction is missing. The correlation between automatic metrics (like MAP) and human preferences needs further investigation.
  • Noise in the dictionary: The Sõnaveeb resource might contain noisy or erroneous synonymy relations or definitions, which could affect evaluation accuracy. Filtering this noise could potentially improve results.
  • User interface and visualization: The paper notes that the visual presentation of search results is crucial for user experience, and this aspect was not evaluated.
  • Future work: The authors plan to develop a user interface for human evaluation and integrate the best-performing model into the Sõnaveeb portal to enhance its functionality for language learners. They also suggest exploring more sophisticated methods for handling noisy data and improving the visual representation of results.

Summary updated

BibTeX
@inproceedings{dorkin2024sonajaht,
  author = {Aleksei Dorkin and Kairit Sirts},
  title = {{Sõnajaht: Definition Embeddings and Semantic Search for Reverse Dictionary Creation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure