TL;DR
This paper introduces novel neural network-based composition functions (CNN and RNN) for learning subword-level word embeddings. By integrating these with a hybrid training scheme, the authors achieve state-of-the-art performance on tasks requiring morphological understanding and handle out-of-vocabulary (OOV) words effectively, while maintaining compact model sizes.
What Problem Does This Paper Address?
Traditional word embedding models, like Skip-Gram and FastText, map entire words to vectors. This approach suffers from two main limitations: data sparsity, where infrequent words are poorly represented or discarded, and a lack of morphological awareness, treating different forms of the same word (e.g., "physicist" and "physicists") as unrelated. While subword models like FastText address OOV words by using character n-grams, this paper explores more sophisticated neural composition functions to better capture morphology and improve overall embedding quality, especially for morphologically rich languages and datasets with high OOV rates.
Methods
The core contribution is the proposal of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) as composition functions to learn subword-level embeddings. These models take sequences of characters (or character n-grams) as input and output a fixed-size vector.
- CNN Approach: Inspired by language modeling architectures, character vectors are first extracted. 1D convolutional filters then capture local features, followed by max-pooling and concatenation.
- RNN Approach: Specifically, Long Short-Term Memory (LSTM) and its bidirectional variant are used. The RNN processes character vectors sequentially, and its hidden states are combined to form the word embedding.
- Hybrid Training Scheme: To leverage both morphological and semantic information, the authors propose training their subword-level models jointly with a traditional word-level lookup-table model. This allows the subword models to benefit from the semantic richness of word-level embeddings, while the word-level embeddings can be discarded after training, leaving only compact subword representations.
- FastText (Summation): This is presented as a baseline subword model, where word embeddings are formed by simply summing the vectors of its constituent character n-grams.
Datasets
The paper uses several datasets for evaluation:
- Training: TEXT8 corpus (first 109 bytes of English Wikipedia dump).
- Word Similarity: Rare Words, WordSim353 (similarity and relatedness), Sim 999, MEN, Mech Turk.
- Word Analogy: Google analogy dataset, BATS analogy dataset.
- Affix Prediction: A dataset from Lazaridou et al. (2013) containing stem-derived word pairs.
- Sequence Labeling: CoNLL 2000 (POS Tagging), CoNLL 2003 (Chunking, NER).
- Text Classification: Movie Review (MR), Customer Reviews (CR), Subjectivity/Objectivity (SUBJ), IMDB movie review.
Key Insights and Ideas
- Neural Compositionality Outperforms Simple Summation: CNN and RNN-based composition functions significantly outperform the simple summation approach of FastText on tasks related to morphology and OOV words. This suggests that more complex neural architectures can better learn the compositional structure of words.
- Hybrid Training is Beneficial: Combining subword-level models with traditional word-level embeddings during training improves performance across various tasks, especially for semantic tasks. This hybrid approach effectively bridges the gap between morphological and semantic representations.
- Compactness and Efficiency: The proposed neural subword models are remarkably compact, requiring only megabytes of parameters. This allows for on-demand generation of embeddings, a significant advantage over large, pre-trained word embedding files.
- Morphology is Key for OOV and Specific Tasks: Subword-level models, particularly CNN and RNN, excel at tasks involving morphology (e.g., word analogy, affix prediction) and demonstrate the critical need for handling OOV words, especially in morphologically rich languages.
- FastText's Limitation: While FastText is a strong subword model, its simple summation approach limits its ability to capture complex morphological relationships, as evidenced by its performance on affix prediction.
Key Results
- Word Analogy: Subword-level models achieved minimal 57.8% accuracy (excluding lookup table versions) on derivation morphology categories, compared to Skip-Gram's 9.6%. With vocabulary expansion, this accuracy reached 82.4%. Concat subword+OOV achieved 96.2% and 96.0% on inflection and derivation morphology, respectively.
- Affix Prediction: CNN and RNN models showed strong performance in predicting affixes, with CNN demonstrating an advantage in distinguishing derivation types.
- Sequence Labeling: Expanding vocabulary with subword-level models provided significant gains. For instance, on NER tasks, subword+OOV models achieved 1.9-3.2% absolute gains over their non-OOV counterparts.
- Word Similarity: FastText models generally performed best, but CNN and RNN subword models were more focused on morphology and thus less effective on pure semantic similarity tasks.
- Model Size: CNN and RNN models required only a few megabytes of data, significantly less than FastText which requires hundreds of megabytes for its character n-gram vocabulary.
Limitations and Future Work
- Semantic vs. Morphological Trade-off: The paper notes that while subword-level models excel at morphology, they might not capture semantic information as well as traditional word-level models. The hybrid training scheme and model concatenation are proposed as ways to mitigate this.
- Text Classification Performance: The paper observed that input word embeddings did not considerably affect text classification accuracy, making it hard to draw definitive conclusions from this task. This aligns with previous findings suggesting that context types might be more influential than embedding types for this task.
- Future Work: The authors suggest that their compact, model-based embeddings can be combined with other approaches to improve morphological sensitivity. They also highlight the potential for their models to be integrated into larger NLP systems. The source code and pre-trained embeddings have been made publicly available.
Summary updated