TL;DR
This paper explores how to create effective word embeddings when training data is scarce (millions of words instead of billions). They find that character-aware models, which learn from sub-word units, outperform traditional word-based models in these low-resource settings, especially for rare words. Combining different embedding types offers the best performance on small datasets.
What Problem Does This Paper Address?
Modern word embedding models, like Word2Vec and GloVe, rely on massive text corpora (billions of words) to learn meaningful representations of words. However, this approach has two major limitations:
- Low-Resource Languages and Domains: Many languages and specialized fields lack such vast amounts of text data, making it difficult to train effective embeddings.
- Human Language Acquisition: Humans learn language by generalizing from relatively limited exposure. Children, for instance, learn tens of thousands of words from hearing and reading millions of words, and they can often infer the meaning of new words by leveraging existing knowledge. Current word embedding models, which treat words as atomic units, struggle with this generalization, especially for infrequent or out-of-vocabulary (OOV) words.
This paper aims to bridge this gap by investigating embedding models that perform well in low-resource scenarios, mimicking human learning more closely.
Methods
The authors compare four embedding models, all based on the skip-gram objective function but differing in their input granularity:
- Word-level Skip-gram (WL): This is a standard, character-agnostic model. It assigns an embedding to each word in the vocabulary. Out-of-vocabulary words are handled by averaging the embeddings of in-vocabulary words.
- fastText (FT): This model builds upon the word-level skip-gram but represents words as a sum of their character n-gram vectors. This allows it to generate embeddings for unseen words by composing them from familiar n-grams.
- Character-aware Skip-gram (CL): This novel model learns representations for individual characters. It uses a Convolutional Neural Network (CNN) to process character sequences of words, generating word embeddings. This allows character-level information to directly inform word representations.
- Concatenation (CAT): This model fuses the embeddings from WL, FT, and CL by concatenating them. Embeddings from individual models are rescaled to have similar average magnitudes before concatenation.
Datasets
Training Data: The models were trained on standard Wikipedia corpora for English and German. To simulate low-resource scenarios, they also created sub-corpora of 10 million and 100 million characters. These sizes are chosen to represent typical Wikipedia sizes for low-resource languages.
Evaluation Benchmarks: The models were evaluated on word relatedness prediction tasks using:
- WordSim353 (WS353): A standard English benchmark.
- WS353-de: A German version of WordSim353.
- Stanford Rare Word Dataset (RW): An English dataset specifically designed to test performance on rare words.
- GUR350: A German benchmark with more frequent, longer words, leading to a higher proportion of OOV words.
Key Insights and Ideas
- Character Awareness Matters for Rare Words: Even on large corpora, character-aware models (FT and CL) show an advantage over plain word-based models (WL) for infrequent words.
- Low-Resource Advantage: On small corpora (millions of words), character-aware models consistently outperform skip-grams, demonstrating their effectiveness in data-scarce environments.
- fastText's Versatility: The fastText model (FT) performs well across various dataset sizes, often matching or exceeding the word-level model, and is a strong candidate for both medium and large datasets.
- Fusion for Low Data: Combining different embedding types (CAT) yields the best results on small corpora, suggesting that WL, FT, and CL capture complementary information.
- Character N-grams vs. Individual Characters: While FT (character n-grams) is generally strong, the CL model (individual characters) shows promise in the lowest-data setups, potentially indicating that modeling individual character embeddings is more suitable when data is extremely limited.
Key Results
- Full Corpora: FT performs as well as or better than WL. CL does not show an advantage on its own, but the fused CAT model is robust and performs on par with WL.
- 100M Corpora: Model performance generally decreases with smaller corpora, but this effect is more pronounced for WL than for FT and CL. CAT outperforms FT on this dataset size, highlighting complementarity.
- 10M Corpora: A more heterogeneous picture emerges. WL performs poorly on OOV pairs. FT outperforms WL on most benchmarks. CL excels on GUR350, suggesting individual character embeddings might be better for the absolute lowest-data scenarios. The fused CAT model significantly outperforms FT on 3 out of 4 benchmarks.
Limitations and Future Work
Limitations: * The study primarily uses English and German, which are not truly low-resource languages, but rather serve as a testbed for controlled low-resource scenarios. * The paper acknowledges that the "full" corpora used are still substantial (billions of words), and the truly extreme low-resource scenarios (thousands or tens of thousands of words) were not explicitly tested.
Future Work: * Evaluate the character-based models for other languages. * Further investigate the psycholinguistic plausibility of these models, such as matching human performance and acquisition speed. * Explore other aspects of character-based representations, like their support for morpheme-based mental lexicon theories.
Summary updated