Introduction
Word embeddings—dense vector representations that capture semantic and syntactic relationships between words—have become the cornerstone of modern Natural Language Processing (NLP). Despite their ubiquity, selecting the optimal embedding model for a specific language and task remains a challenge. Most researchers rely on "intrinsic" evaluations, such as word analogy tests (e.g., "king is to queen as man is to woman"), because they are computationally inexpensive. However, there is growing skepticism regarding whether performance on these abstract analogies translates to success in real-world "extrinsic" applications.
This study addresses this gap for the Portuguese language by conducting a large-scale evaluation of 31 different word embedding models. The researchers investigate two primary questions: how do various architectures perform on Portuguese-specific data, and does a model's ability to solve analogies actually predict its utility in downstream tasks like Part-of-Speech (POS) tagging and semantic similarity?
Methodology and Data
The researchers trained 31 models using a massive, multi-genre corpus of approximately 1.39 billion tokens. This training set combined European Portuguese (LX-Corpus) and Brazilian Portuguese (Wikipedia, news portals like G1, and literary works) to ensure broad linguistic coverage.
Four popular embedding architectures were evaluated across varying vector dimensions (50, 100, 300, 600, and 1,000):
- Word2Vec (CBOW and Skip-Gram): The foundational predictive models.
- GloVe: A count-based model focusing on global word co-occurrence statistics.
- FastText: An evolution of Word2Vec that incorporates subword information (character n-grams), theoretically beneficial for morphologically rich languages like Portuguese.
- Wang2Vec: A modification of Word2Vec designed to account for word order, a feature often ignored by standard architectures.
Evaluation was split into intrinsic tests (a Portuguese adaptation of the Mikolov analogy dataset) and extrinsic tests, including POS tagging (using the Mac-Morpho corpus) and semantic similarity (using the ASSIN dataset).
The Analogy Paradox
The results revealed a significant "evaluation gap." As shown in the intrinsic evaluation table, GloVe models generally excelled at semantic analogies. For instance, GloVe with 600 dimensions achieved the highest semantic scores for both Brazilian (48.5) and European (43.8) Portuguese variants.
However, this dominance did not translate to downstream tasks. GloVe performed poorly in POS tagging and sentence similarity compared to predictive models. This confirms that analogy tests are a poor proxy for a model's practical utility in complex NLP pipelines. Interestingly, while FastText was expected to dominate due to its subword awareness, it primarily led in syntactic analogies rather than the more complex POS tagging task.
Downstream Performance and Word Order
In extrinsic evaluations, the importance of word order became evident. Wang2Vec, which explicitly models the sequence of words, emerged as the superior choice for POS tagging. The Wang2Vec Skip-Gram model with 1,000 dimensions achieved a peak accuracy of 95.94%.
For semantic similarity, results varied by dialect. For Brazilian Portuguese, Wang2Vec Skip-Gram (1,000 dimensions) led with a Pearson correlation of 0.60. For European Portuguese, Word2Vec CBOW (1,000 dimensions) proved most effective. The study also noted diminishing returns regarding vector size; while increasing dimensions from 50 to 300 typically yielded large gains, moving to 1,000 dimensions often resulted in plateaus or slight regressions in performance.
Conclusion and Future Directions
The study highlights that for Portuguese NLP, the choice of embedding model should be driven by the target task rather than analogy benchmarks. The researchers have publicly released all 31 models and their preprocessing scripts to support the community.
Future work will address current limitations, such as the handling of clitics (e.g., "-se" in "machucou-se"), which currently contribute to high Out-of-Vocabulary (OOV) rates. Exploring lemmatization and expanding evaluations to tasks like Named Entity Recognition (NER) and Sentiment Analysis will further refine our understanding of Portuguese word representations.
Summary updated