Abstract: Tomáš Musil, Jonáš Vidra, David Mareček. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
2019.
Introduction
This research investigates whether word embeddings—trained on whole words without explicit sub-word information—inherently capture derivational morphology. By clustering the difference vectors of word pairs in the morphologically rich Czech language, the authors demonstrate that these embeddings successfully group words into semantic categories like "actor," "diminutive," and "nominalization" without any explicit supervision.
The Challenge of Derivation
In linguistics, derivation is the process of forming new words from existing ones (e.g., "collision" from "collide"). While modern NLP often relies on sub-word units (like BPE or WordPiece) to handle rare words, this paper asks a fundamental question: Do standard whole-word embeddings already "know" about these relationships?
The challenge is particularly acute in morphologically rich languages like Czech, where a single root can sprout a massive family of related words. The authors explore whether the "vector offset" hypothesis—famously illustrated by the $king - man + woman = queen$ analogy—holds true for complex morphological transformations.
Methodology
The researchers employed an unsupervised approach to analyze vector spaces:
- Vector Differences: For a pair of words $(W_1, W_2)$ where $W_2$ is derived from $W_1$, they calculated the difference vector $d = v_2 - v_1$.
- Unsupervised Clustering: These difference vectors were processed using three algorithms: K-means, Hierarchical Agglomerative Clustering (Euclidean), and Hierarchical Agglomerative Clustering (Cosine).
- Normalization: Results were compared using both raw vectors and vectors normalized to unit length.
- Model Comparison: The study tested embeddings from a standard word2vec (skip-gram) model and several Neural Machine Translation (NMT) architectures, including RNNs and Transformers.
To evaluate the clusters, 71 frequent derivation types were manually mapped into 21 semantic classes.
Datasets
- DeriNet 1.6: A Czech lexical network containing over one million lemmata organized into derivational trees.
- Czech National Corpus (syn 4): A 4.6-billion-token corpus used to train the word2vec models.
- CzEng 1.6: A large parallel Czech-English corpus (78 million tokens) used to train the NMT models.
Results and Insights
The study reveals that derivational relations are represented as consistent directions in embedding space. Key findings include:
- word2vec Performance: The word2vec model performed best, achieving a V-measure of 61.59 and an accuracy of 77.0% when clustering into 21 groups.
- Normalization is Crucial: Normalizing vectors to unit length significantly improved clustering performance, likely because it prevents word frequency (magnitude) from interfering with morphological direction.
- Semantics Over Syntax: The models capture semantic shifts rather than just grammatical category changes. For instance, "perfectivization" (verb to verb) was identified with 92.85% precision.
- High Precision Categories: Certain categories were nearly perfectly identified, such as "man → woman" (98.01% precision) and "instrument/scientist" (97.46%).
- Architecture Differences: In NMT models, decoders captured more derivational information than encoders, likely because the decoder acts partially as a language model. Interestingly, plain RNNs outperformed Transformers in storing relational info, possibly because Transformers "spread" information across parameters via attention and residual connections.
Limitations and Future Work
The study was constrained by the 25,000-word vocabulary of the NMT models and the tree-based simplification of DeriNet. Future research could compare these results against sub-word models (like FastText) or extend the analysis to contextualized embeddings like BERT to see how derivational relations shift based on usage context.
Summary updated