Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks

Nathan Hartmann, Erick Rocha Fonseca, Christopher Shulby, Marcos V. Treviso, Jéssica S. Rodrigues, Sandra M. Aluísio

Abstract

This study evaluates 31 word embedding models for the Portuguese language using popular algorithms including Word2Vec, GloVe, FastText, and Wang2Vec. By comparing performance on traditional word analogies against real-world NLP tasks such as POS tagging and semantic similarity, the authors identify a significant disconnect: models that excel at analogies often underperform on actual downstream tasks. The research recommends Wang2Vec for Portuguese NLP and suggests that embedding dimensions beyond 300 offer diminishing returns.

Introduction

Word embeddings are a foundational component of modern Natural Language Processing (NLP), but evaluating their quality remains a challenge. Traditionally, researchers use intrinsic benchmarks like word analogies (e.g., "king is to queen as man is to woman"). However, recent studies suggest these analogies might not correlate with how well the embeddings actually perform in downstream applications. This paper tackles this evaluation gap specifically for the Portuguese language, encompassing both Brazilian and European variants, to determine which algorithms work best and whether standard analogy tests are reliable proxies for real-world performance.

Methodology

The researchers trained 31 distinct word embedding models using four popular algorithms: GloVe, Word2Vec (CBOW and Skip-Gram), Wang2Vec (a modification of Word2Vec that accounts for word order), and FastText (which represents words as sums of character n-grams). Models were trained across various dimensionalities (50, 100, 300, 600, and 1,000) on a massive, multi-genre corpus of approximately 1.4 billion tokens combining sources like Wikipedia, news portals, and literary works.

Results and Analysis

Intrinsic vs. Extrinsic Performance

The most striking finding is the lack of correlation between intrinsic and extrinsic evaluations. While some models dominated the analogy tests, they failed to maintain that lead in practical applications.

Intrinsic evaluation results for word analogies across different models and dimensions.

As shown in the intrinsic evaluation table above, GloVe (600d) achieved the highest accuracy on semantic analogies (48.5% for Brazilian Portuguese), while FastText (300d Skip-Gram) led in syntactic analogies (58.7%). However, when these models were applied to the extrinsic task of Part-of-Speech (POS) tagging, the rankings shifted significantly.

Extrinsic evaluation results showing POS tagging accuracy for all tested models.

In the POS tagging task, Wang2Vec (300d Skip-Gram) achieved the highest accuracy (95.89%), outperforming even 1000d Word2Vec models. Conversely, GloVe and FastText—the leaders in analogy benchmarks—performed the worst in this downstream application.

Key Insights

  • Word Order Matters: Wang2Vec's success suggests that respecting word order is crucial for capturing the nuances of Portuguese semantics and syntax.
  • Morphology is Not a Silver Bullet: Despite explicitly modeling subword morphology, FastText yielded relatively poor results on POS tagging, a task traditionally reliant on morphological cues.
  • Diminishing Returns on Dimensionality: Pushing embedding dimensions beyond 300 generally provided only marginal improvements. In some cases, performance even dropped at higher dimensions, suggesting that 300d is the optimal balance between accuracy and memory usage.

Conclusion

The study concludes that analogy benchmarks are a misleading metric for selecting word embeddings for real-world tasks. For Portuguese NLP, Wang2Vec is the recommended model. Future work will explore different tokenization and normalization patterns, such as lemmatizing verbs, to improve processing efficiency and handle complex linguistic structures like clitic pronouns.

Summary updated

BibTeX
@inproceedings{hartmann2017portuguese,
  author = {Nathan Hartmann and Erick Rocha Fonseca and Christopher Shulby and Marcos V. Treviso and Jéssica S. Rodrigues and Sandra M. Aluísio},
  title = {{Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks}},
  booktitle = {Stil},
  year = {2017},
}
Figure