Derivational Morphological Relations in Word Embeddings

Tomás Musil, Jonás Vidra, David Marecek

Abstract

Derivation is a type of a word-formation process which creates new words from existing ones by adding, changing or deleting affixes. In this paper, we explore the potential of word embeddings to identify properties of word derivations in the morphologically rich Czech language. We extract derivational relations between pairs of words from DeriNet, a Czech lexical network, which organizes almost one million Czech lemmata into derivational trees. For each such pair, we compute the difference of the embeddings of the two words, and perform unsupervised clustering of the resulting vectors. Our results show that these clusters largely match manually annotated semantic categories of the derivational relations (e.g. the relation 'bake--baker' belongs to category 'actor', and a correct clustering puts it into the same cluster as 'govern--governor').

Introduction

In linguistics, derivation is the process of forming new words from existing ones, typically by adding prefixes or suffixes (e.g., "teach" becomes "teacher"). While word embeddings are renowned for capturing linear analogies like king - man + woman = queen, their ability to represent the structured, hierarchical nature of word families has been less explored. This study investigates whether standard word embeddings implicitly capture the semantic relationships between base words and their derivatives in Czech, a language with a complex morphological system.

Example of a derivational tree from DeriNet for the root "živit" (to nourish).

Methodology

The researchers employed an unsupervised approach to evaluate how well vector differences represent semantic changes:

  1. Relation Extraction: Pairs of base and derived words were extracted from DeriNet 1.6, a Czech lexical network containing over 1 million lemmata and 800,000+ relations.
  2. Vector Arithmetic: For each pair $(W_1, W_2)$, a difference vector $d = v_2 - v_1$ was calculated.
  3. Clustering: Three algorithms—K-means, Hierarchical Agglomerative Clustering (Euclidean), and Hierarchical Agglomerative Clustering (Cosine)—were used to group these difference vectors.
  4. Evaluation: Clusters were compared against 21 manually annotated semantic classes (e.g., "actor," "diminutization") using metrics such as V-measure and accuracy.

The study compared embeddings from word2vec (trained on the Czech National Corpus) and internal embeddings from three Neural Machine Translation (NMT) architectures: RNN, RNN with attention, and Transformer (trained on the CzEng 1.6 corpus).

Key Insights

The Superiority of word2vec

A primary finding was that word2vec embeddings outperformed NMT-based embeddings in capturing derivational semantics. This suggests that word2vec’s training objective is particularly effective at learning broad relational properties compared to task-specific NMT embeddings.

Comparison of different embedding models across clustering metrics (H: Homogeneity, C: Completeness, V: V-measure, A: Accuracy).

Architectural Impact and Complexity

Within NMT models, the Transformer—despite being the most complex architecture—performed the worst. The authors hypothesize that the Transformer's high parameter count and residual connections spread "meaning" across the network, making the initial embedding layer less informative for this specific task than in simpler RNN models. Additionally, decoder embeddings generally captured more relational information than encoder embeddings, likely because the decoder functions more similarly to a traditional language model.

The Role of Normalization

Normalizing word vectors to unit length before clustering was found to be essential. Without normalization, clustering algorithms often failed to identify meaningful semantic patterns, indicating that the magnitude of the vectors can interfere with the relational signal.

Results and Discussion

The system achieved a peak accuracy of 77.00% using K-means on normalized word2vec vectors, significantly outperforming a random baseline (~31%). Certain categories were identified with exceptional precision, such as man → woman (98.01%) and perfectivization (92.85%).

However, some categories proved challenging. Diminutization (e.g., "house" → "little house") was frequently confused with "man → woman" relations, suggesting these processes might share a similar semantic axis in the vector space. Furthermore, the model's performance was often dependent on Part-of-Speech (POS) pairs, with cross-POS categories like "super+" being harder to cluster.

Conclusion

This research demonstrates that word embeddings can implicitly learn the semantic structure of word derivations without explicit subword information. These findings suggest that embeddings could be used to semi-automatically expand or verify lexical networks like DeriNet. Future work may explore how modern subword-aware models represent the semantic vs. formal aspects of derivation.

Summary updated

Abstract

Derivation is a type of a word-formation process which creates new words from existing ones by adding, changing or deleting affixes. In this paper, we explore the potential of word embeddings to identify properties of word derivations in the morphologically rich Czech language. We extract derivational relations between pairs of words from DeriNet, a Czech lexical network, which organizes almost one million Czech lemmata into derivational trees. For each such pair, we compute the difference of the embeddings of the two words, and perform unsupervised clustering of the resulting vectors. Our results show that these clusters largely match manually annotated semantic categories of the derivational relations (e.g. the relation 'bake--baker' belongs to category 'actor', and a correct clustering puts it into the same cluster as 'govern--governor').
BibTeX
@inproceedings{musil2019derivational,
  author = {Tomás Musil and Jonás Vidra and David Marecek},
  title = {{Derivational Morphological Relations in Word Embeddings}},
  booktitle = {BlackboxNLP @ ACL},
  year = {2019},
  url = {https://openreview.net/forum?id=dcnCsutlGI},
}
Figure