TL;DR
This paper investigates how to improve word embeddings for Hungarian, a morphologically rich language with flexible word order, which poses challenges for standard embedding techniques. They find that using lemmas (dictionary forms) of words significantly boosts semantic accuracy, while character n-grams, surprisingly, degrade performance.
What Problem Does This Paper Address?
Word embeddings, like those generated by Word2Vec or fastText, have revolutionized Natural Language Processing (NLP) by capturing semantic relationships between words. However, these methods often struggle with morphologically rich languages like Hungarian. In such languages, words can have many different forms due to suffixes and prefixes (agglutination), leading to a vast vocabulary and increased variance in word occurrences. This makes it harder for embeddings to learn consistent representations. Furthermore, Hungarian's free phrase order means a word can appear in many different contexts, further complicating the learning process. The paper aims to find sub-word embedding strategies that can overcome these challenges and improve semantic accuracy for Hungarian.
Methods
The authors explore several sub-word embedding strategies using the fastText tool, which allows for character n-gram augmentation. Their main strategies include:
- Word Vectors (W): The baseline approach using standard fastText with whole words.
- Lemma Vectors (L): Utilizing lemmas derived from a morphological analyzer (magyarlánc) as the embedding units. This means instead of embedding "házak" (houses), they embed "ház" (house).
- Morf Vectors (M) and Root Vectors (R): Employing unsupervised segmentation of words into "morfs" (sub-word units) or roots using the Morfessor tool.
They systematically evaluate these strategies by varying:
- Embedding Dimension: Testing 100 and 200 dimensions.
- Context Window Size: Ranging from 5 to 25 words.
- Character N-gram Augmentation: Comparing embeddings with and without character n-grams.
Datasets
The paper uses a custom corpus of Hungarian web pages from 2014-2018, comprising over 70 million word tokens. For evaluation, they employ a Hungarian analogy test (Makrai, 2015) designed according to the standard Mikolov et al. (2013a) methodology. This test includes semantic tasks (country-capital, country-currency, family) and syntactic tasks (singular-plural, present-past, base-comparative).
Key Insights and Ideas
- Lemmas are King for Semantics: The most significant finding is that using lemmas (dictionary forms) of words as embedding units dramatically improves semantic accuracy in Hungarian. This suggests that normalizing words to their base form helps disambiguate meaning and create more consistent representations.
- Character N-grams Backfire: Counter-intuitively, adding character n-grams, which are often beneficial for handling rare words and improving robustness in other languages, actually decreases semantic accuracy in Hungarian. The authors hypothesize this is due to the high degree of agglutination, making word vectors too "universal" and unable to specialize for context.
- Longer Context Windows Help Semantics, Hurt Syntax: Extending the context window size generally improves semantic accuracy, likely by allowing the model to capture more distant but relevant words. However, this also tends to decrease syntactic accuracy, possibly due to increased noise and variance in the larger context.
- Morphosyntax is Learned Well: Even with these challenges, Hungarian embeddings are shown to learn morphosyntax quite well.
Key Results
- Lemma vectors (L) achieved the highest semantic accuracy, outperforming the baseline word vectors (W) by up to 75%.
- Adding character n-grams consistently resulted in decreased semantic accuracy across all strategies.
- Increasing the embedding dimension from 100 to 200 led to a positive effect on semantic accuracies, with up to a 50% relative increase.
- Longer context windows (up to 21 units) showed a consistent increase in semantic accuracy for all strategies.
Limitations and Future Work
The authors acknowledge that their study focuses on specific sub-word strategies and the fastText tool. They suggest future work could:
- Explore other embedding algorithms: Investigate Word2Vec, GloVe, and other models.
- Extend dimensionality analysis: Test higher embedding dimensions.
- Analyze other morphologically rich languages: Apply these findings to languages like Finnish, Turkish, or Slavic languages.
- Use more challenging evaluation sets: The current analogy test might not fully capture lexical semantic relations or world knowledge. They point to more difficult test sets that could be translated for richer evaluation.
Summary updated