TL;DR
This paper presents a method for training high-quality word vectors for 157 languages by combining Wikipedia and Common Crawl data. They also introduce new word analogy datasets for French, Hindi, and Polish, and demonstrate that their models achieve strong performance across various languages, especially for those with limited Wikipedia resources.
What Problem Does This Paper Address?
Word vectors, or distributed representations of words, are fundamental to many Natural Language Processing (NLP) tasks, enabling models to understand word meanings and relationships. The quality of these vectors heavily relies on the size and quality of the training data. While pre-trained word vectors are widely available for English, high-quality, multilingual models have been scarce. Existing methods often rely on Wikipedia, which, despite its quality, can be small for many languages. This paper addresses the challenge of creating robust, multilingual word vectors by leveraging both the curated content of Wikipedia and the vast, albeit noisier, data from the Common Crawl.
Methods
The authors employ an extension of the fastText model, which incorporates subword information. This means that instead of just learning vectors for whole words, fastText also learns vectors for character n-grams. A word's vector is then derived from the sum of its constituent character n-gram vectors, allowing it to handle out-of-vocabulary words and capture morphological similarities.
Their training pipeline involves several key steps:
- Data Collection: They gather data from Wikipedia dumps and the Common Crawl corpus.
- Language Identification: A fast, custom language detector is trained using fastText's linear classifier to accurately identify the language of text lines from the Common Crawl. Lines are filtered based on length and confidence score.
- Deduplication: To handle the redundancy in web data, lines with identical hashes are removed.
- Tokenization: Different tokenization tools are used depending on the language's script (e.g., Stanford word segmenter for Chinese, Mecab for Japanese).
- Model Training: The fastText skipgram model with subword information is used to train word vectors on the preprocessed data. They also explore the CBOW (Continuous Bag-of-Words) model with positional weights and subword information.
Datasets
- Training Data:
- Wikipedia: Used for its high-quality, curated content.
- Common Crawl: A large-scale, noisy web crawl dataset, used to augment training data, especially for languages with smaller Wikipedia corpora.
- Evaluation Datasets:
- Existing Analogy Datasets: For Czech, German, Spanish, Finnish, Italian, Portuguese, and Chinese.
- Newly Introduced Datasets: * French: A dataset of 31,688 analogy questions, including translated existing categories and new ones like "capitals of French départements" and "antonyms-adjectives." * Hindi: Analogies adapted from English, including specific Hindi cultural nuances like maternal/paternal grandparents and city-state pairs from India. Categories that don't translate well (e.g., adjective-to-adverb) were removed or adapted.
- Polish: Analogies covering capital-common-countries, capital-world, currency, Polish regions (województwo), and verb-aspect pairs.
Key Insights and Ideas
- Subword Information is Crucial: The paper reinforces the importance of subword information (character n-grams) in fastText for handling morphology and out-of-vocabulary words, especially in morphologically rich languages.
- Common Crawl for Low-Resource Languages: Combining Wikipedia with the Common Crawl significantly boosts performance for languages with small Wikipedia corpora, demonstrating the value of large, noisy web data when high-quality data is scarce.
- CBOW with Positional Weights Outperforms: The CBOW model, when enhanced with positional weights and subword information, shows the most significant improvement in analogy task performance.
- Hyperparameter Tuning Matters: Increasing the number of negative samples and training epochs leads to substantial accuracy gains, albeit at the cost of longer training times.
Key Results
- The models trained on Wikipedia and Common Crawl achieve high accuracy on word analogy tasks across 10 languages.
- For languages with small Wikipedia resources (e.g., Finnish, Hindi, Polish, Chinese), using Common Crawl data leads to substantial improvements in accuracy (e.g., +23.5% for Finnish, +16.0% for Hindi).
- For high-resource languages (German, Spanish, French), the addition of Common Crawl data either has a negligible effect or slightly decreases accuracy, likely due to the analogy datasets' domain alignment with Wikipedia.
- The authors' custom language detector is significantly faster than existing tools like
langid.pywhile maintaining comparable accuracy.
Limitations and Future Work
- Noisy Data Impact: While the Common Crawl is valuable, its noisy nature might limit performance on tasks that are highly sensitive to data quality, especially for high-resource languages where Wikipedia is already extensive.
- Domain Specificity: The analogy datasets used for evaluation are largely aligned with encyclopedic content. This means the models' performance might differ on analogy tasks from other domains.
- Future Work: The authors suggest exploring more techniques to further improve model quality, particularly for low-resource languages. They also plan to release their models and language identifier for public use.
Summary updated