Experiments on a Guarani Corpus of News and Social Media

Santiago Góngora, Nicolás Giossa, Luis ChiruzzoNAACL 2021

TL;DR

This paper introduces a new, large-scale Guarani-Spanish parallel corpus and a monolingual corpus of Guarani tweets, addressing the scarcity of resources for this widely spoken but under-resourced language. Experiments using word embeddings on this new corpus show promising results for NLP tasks, highlighting the need for diverse text domains to further improve performance.

What Problem Does This Paper Address?

Guarani is a significant indigenous language spoken by millions across South America, particularly in Paraguay, where it holds official status. Despite its widespread use, the availability of digital text resources for Guarani, especially for NLP applications like machine translation, is severely limited. This scarcity hinders research and development in the field. Existing resources are often small, not easily downloadable, or focus on specific dialects. This paper aims to bridge this gap by constructing a comprehensive Guarani corpus that reflects the language's real-world usage, including both formal news articles and informal social media content, to support future NLP advancements.

Methods

The authors employed a multi-pronged approach to corpus construction. For the parallel news corpus, they utilized a web crawler to gather news articles from Paraguayan websites (.py domain). They initiated this process by querying search engines with known Guarani words and then followed internal links to expand their collection. To match Guarani and Spanish articles, they developed a heuristic based on timestamps, grouping articles by creation date and pairing those with the closest timestamps. For the monolingual tweet corpus, they initially attempted to use the Twitter API's language detection, but found it ineffective for Guarani. Consequently, they trained their own Naïve Bayes classifier for language detection, which, while good on cleaner text, struggled with the noisy nature of tweets. Ultimately, they adopted a frequent words-based approach, creating lists of common Guarani words to identify relevant tweets. They then categorized these tweets into "very reliable," "reliable," and "unreliable" based on the number of identified Guarani words and their origin (local vs. global), using the "reliable" categories for their corpus.

Datasets

The paper introduces two new datasets:

  • Parallel Guarani-Spanish News Corpus: This corpus consists of 2,580 news articles, resulting in 14,792 sentence pairs. It includes 334,501 Guarani word tokens and 635,226 Spanish word tokens. This is a significant expansion compared to previous parallel corpora.
  • Monolingual Guarani Tweets Corpus: This corpus is categorized into three levels of reliability:
  • Very Reliable: 532 tweets, 7,706 tokens.
  • Reliable: 4,199 tweets, 48,895 tokens.
  • Unreliable: 46,197 tweets, 453,996 tokens. The "reliable" tweets (categories A and B) were combined with Guarani sentences from the parallel set to form the monolingual split.

The authors also utilized existing datasets for experiments:

  • Guarani Wikipedia text: Used for training word embeddings.
  • Guarani data from Chiruzzo et al. (2020): A previously established parallel corpus, used for comparison and training.

Key Insights and Ideas

  • The challenge of web crawling for under-resourced languages: Even with a specific top-level domain (.py), most web content defaults to the majority language (Spanish), making it difficult to find sufficient text in the target language (Guarani).
  • Jopara's mixed nature: The Jopara dialect, spoken in Paraguay, exhibits varying degrees of Spanish influence, making it a complex but important target for NLP resources.
  • Limitations of automated language detection: Standard APIs are often insufficient for detecting less common languages or specific dialects, necessitating custom solutions.
  • The trade-off between corpus size and noise: While social media provides a vast amount of text, its inherent noise can negatively impact the quality of NLP models, as seen with word embeddings.
  • Semantic clustering in word embeddings: Visualizations of word embeddings demonstrate that related words, even from different semantic categories, tend to cluster together, indicating that the learned representations capture meaningful relationships.

Key Results

  • The constructed parallel corpus contains 14,792 Guarani-Spanish sentence pairs, a substantial increase over previous efforts.
  • Word embedding experiments showed that the new corpus, when combined with other Guarani text sources, improved performance on analogies tests compared to using only Wikipedia text.
  • Visualizations of word embeddings revealed distinct clusters for categories like countries, animals, and colors, suggesting that the embeddings capture semantic relationships.
  • The "family" analogies test showed that while the new corpus improved exact matches, it did not outperform previous results as much as other categories, suggesting a need for more diverse text types, particularly those that capture familial relationships.
  • Including noisy tweet data in word embedding training generally hindered performance, suggesting that careful filtering or more advanced techniques are needed to leverage social media text effectively.

Limitations and Future Work

The authors acknowledge several limitations and outline future directions:

  • Diversity of text domains: The current corpus, while improved, still benefits from greater diversity. The authors plan to continue crawling for more text and explore additional data sources.
  • Noisy tweet data: The performance degradation observed when including unreliable tweets suggests that more sophisticated methods are needed to extract valuable information from this noisy data.
  • Machine Translation experiments: The authors plan to use the constructed corpus for machine translation experiments, which they believe will be a more robust way to validate the dataset's utility.
  • Statistical approaches for language detection: With more annotated data, they aim to move beyond keyword-based strategies for tweet language detection towards more robust statistical models.
  • Sentiment analysis: The authors believe the corpus, especially the social media component, could be valuable for tasks like sentiment analysis, offering insights into modern Guarani usage.

Summary updated

BibTeX
@inproceedings{gongora2021experiments,
  author = {Santiago Góngora and Nicolás Giossa and Luis Chiruzzo},
  title = {{Experiments on a Guarani Corpus of News and Social Media}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure