Abstract
Linguistic similarity is multi-faceted. For instance, two words may be similar with respect to semantics, syntax, or morphology inter alia. Continuous word-embeddings have been shown to capture most of these shades of similarity to some degree. This work considers guiding word-embeddings with morphologically annotated data, a form of semi-supervised learning, encouraging the vectors to encode a word's morphology, i.e., words close in the embedded space share morphological features. We extend the log-bilinear model to this end and show that indeed our learned embeddings achieve this, using German as a case study.
Introduction
In Natural Language Processing (NLP), word embeddings like Word2Vec map words into a continuous vector space where proximity implies similarity. However, "similarity" is a multi-axial concept that can be topical (ice and cold), syntactic (ice and fire), or morphological (ice and icy). Most popular embedding techniques treat words as atomic units, which is problematic for morphologically rich languages like German, Czech, or Turkish. In these languages, a single word contains complex information about gender, case, and number. Failing to recognize the relationship between Stadt (city, nominative) and Städte (cities, plural) results in a significant loss of structural information.
Methods: The Morph-LBL Model
The researchers propose Morph-LBL, a multi-task extension of the Log-Bilinear Model (LBL). While a standard LBL predicts the next word based on a linear combination of preceding embeddings, Morph-LBL is trained to jointly predict both the next word ($w$) and its morphological tag ($t$).
Key technical features include:
- Multi-task Objective: Optimizing the joint probability distribution $p(w, t | h)$ forces word embeddings to store information useful for both language modeling and identifying morphological features.
- Sub-tag Decomposition: Rather than treating a complex tag (like
N.NOM.SG.FEM) as an opaque label, the model decomposes it into binary features. This allows the model to learn that different words share specific traits. - Semi-Supervised Learning: Recognizing that fully annotated data is rare, Morph-LBL is designed to work with partially labeled corpora. For words without tags, the model marginalizes out the missing information, allowing it to learn from massive amounts of raw text while being "steered" by a smaller set of labeled data.
Experimental Setup
The model was evaluated using the TIGER corpus of German newspaper text. Training utilized 200,000 words, with morphological annotations provided for only the first 100,000 to simulate a semi-supervised environment. Baselines included the vanilla LBL and the Skip-Gram model from Word2Vec (trained on both the TIGER corpus and a larger 45-million-token Europarl dataset).
Results and Discussion
By adding the tagging task, the authors effectively "tilt" the vector space. While standard models might cluster words by topic, Morph-LBL clusters them by structural function.
A key insight is that morphology can be inferred from context—a "contextual signature." For example, in German, an adjective in the genitive case is almost always followed by a noun in the genitive case. Morph-LBL exploits these patterns to extract morphological information even for words that were never seen with an annotation during training.
Quantitative Evaluation
The embeddings were tested by seeing how well a simple classifier could predict a word's morphological tag based solely on its vector. Morph-LBL achieved 81.5% accuracy on the "All Types" test, significantly outperforming vanilla LBL (22.1%) and Word2Vec (10.2%). Notably, for words whose tags were never seen during training, Morph-LBL still achieved 44.8% accuracy, nearly triple the performance of the baselines.
The authors also introduced MorphoSim, a metric measuring the average Hamming distance between the morphological tags of a word and its $k$-nearest neighbors in the embedding space.
As shown in the figure, Morph-LBL (M-LBL) consistently yields much lower distances, meaning its neighbors are far more likely to share the same grammatical structure compared to standard embedding models.
Conclusion
Morph-LBL demonstrates that morphological information can be successfully encoded into word embeddings through a semi-supervised multi-task approach. While the current model is blind to character-level internal structures (e.g., it doesn't "see" the -ing suffix), it proves that contextual signatures are a powerful signal. Future work aims to integrate word-internal features, such as character-level CNNs or LSTMs, to combine contextual power with explicit orthographic cues.
Summary updated