On the logistical difficulties and findings of Jopara Sentiment Analysis

Marvin Agüero-Torales, David Vilares, Antonio López-HerreraNAACL 2021

TL;DR

This paper tackles the challenge of sentiment analysis for Jopara, a code-switching language blending Guarani and Spanish. The authors introduce the first Jopara sentiment analysis dataset, highlighting the difficulties in collecting low-resource data, and compare traditional and deep learning models, finding transformer models perform best despite not being pre-trained on Guarani.

What Problem Does This Paper Address?

Natural Language Processing (NLP) applications often overlook indigenous languages, leading to a scarcity of resources. This is particularly true in Latin America, where languages like Guarani coexist with dominant ones like Spanish. Jopara, a code-switching language that mixes Guarani and Spanish (with occasional Portuguese and English influences), is a prime example of such a low-resource scenario. While sentiment analysis (SA) has seen significant progress in high-resource languages, applying it to Jopara presents unique challenges due to the limited availability of annotated data and the complexities of code-switching. This paper aims to bridge this gap by developing a Jopara sentiment analysis dataset and evaluating various machine learning and deep learning models on this challenging task.

Methods

The authors explored two main approaches for collecting Jopara tweets:

  • Keyword-based Scraping: This initial attempt involved using a list of Guarani keywords to crawl Twitter. However, this method yielded a very small number of relevant tweets, demonstrating the difficulty of relying solely on keywords for low-resource languages.
  • Account-based Scraping: A more successful approach involved identifying and scraping tweets from Twitter accounts that predominantly tweet in Guarani. This method, combined with careful filtering and annotation, proved more effective in building a usable dataset.

For sentiment analysis, they compared several models:

  • Traditional Machine Learning: Naïve Bayes (specifically Complement Naïve Bayes to address assumption issues) and Support Vector Machines (SVMs) were used as baselines. These models were trained using TF-IDF weighted unigram and bigram representations.
  • Deep Learning Models:
    • Recurrent Neural Networks (RNNs): Long Short-Term Memory (LSTM) networks were employed, processing tweets as sequences of token vectors.
  • Convolutional Neural Networks (CNNs): CNNs were also used for sentiment analysis, a technique known for its effectiveness in text classification.
  • Transformer-based Models: Pre-trained language models like BERT (specifically Spanish BERT, "beto-base-uncased") and multilingual BERT (mBERT) were fine-tuned on the Jopara dataset. These models leverage wordpiece tokenization and contextual embeddings.

Datasets

The primary dataset introduced in this paper is JOSA (Jopara Sentiment Analysis dataset).

  • Collection: JOSA was created by scraping tweets from user accounts that primarily tweet in Guarani.
  • Size: The initial collection yielded over 23,000 tweets, which were then filtered for similarity and language identification, resulting in 3,948 tweets.
  • Annotation: The dataset was annotated by two native Guarani and Spanish speakers for language (Guarani, Jopara, or other) and sentiment (positive, neutral, negative).
  • Final Size: After filtering for Guarani-dominant tweets, the final dataset comprises 3,491 tweets.
  • Versions: Both unbalanced and balanced versions of JOSA were created for experiments. The unbalanced version reflects real-world distribution, while the balanced version aims to mitigate class imbalance.
  • Splits: For both versions, the data was split into training (50%), development (10%), and test (40%) sets.

Key Insights and Ideas

  • Data Scarcity is a Major Hurdle: The paper strongly emphasizes the significant logistical difficulties in collecting quality data for low-resource languages like Jopara, even for seemingly straightforward tasks like sentiment analysis. Simple keyword-based approaches are often insufficient.
  • Code-Switching Complexity: Jopara's mixed nature, with Guarani and Spanish words, presents challenges for language identification and sentiment analysis. Words with identical forms in both languages and loanwords further complicate the task.
  • Transformer Models Shine: Despite not being pre-trained on Guarani, transformer-based models (specifically Spanish BERT) achieved the best performance. This suggests their inherent cross-lingual capabilities are highly beneficial, even when the target language is not explicitly included in the pre-training data.
  • Traditional Models Remain Competitive: Surprisingly, traditional machine learning models like Naïve Bayes and SVMs performed closely to deep learning models. This is attributed to the low-resource nature of the task, where the limited data might not fully leverage the power of complex deep learning architectures.
  • Complementarity of Models: The error analysis revealed that different models exhibit distinct tendencies in misclassifications. This suggests that ensemble methods, combining the strengths of various models, could be a promising avenue for improving performance in low-resource settings.

Key Results

  • Dataset Size: The JOSA dataset contains 3,491 tweets for sentiment analysis.
  • Model Performance:
    • Best Performer: Spanish BERT ("beto-base-uncased") achieved the highest accuracy on both the unbalanced (0.64) and balanced (0.64) JOSA datasets.
  • Competitive Baselines: Traditional models like SVM achieved accuracies of 0.55 (unbalanced) and 0.54 (balanced).
  • Deep Learning (non-transformer): The best performing BiLSTM-based model achieved 0.49 (unbalanced) and 0.53 (balanced).
  • Neutral Class Dominance: The dataset exhibits a tendency towards neutral tweets, likely due to the nature of content from the scraped accounts (news, announcements, etc.).
  • Model Tendencies: CNB tended to over-classify tweets as negative, while deep learning models were better at identifying neutral tweets. BETO showed the best performance in identifying positive tweets.

Limitations and Future Work

  • Limited Data Size: The JOSA dataset, while the first of its kind, is still relatively small, which impacts the performance of complex models.
  • Topic Diversity: The scraped accounts limited the diversity of topics covered in the dataset.
  • Neutral Tweet Bias: The dataset has a notable imbalance towards neutral tweets.
  • Future Work:
    • Ensemble Methods: The authors suggest exploring ensemble methods to combine complementary models for improved performance.
  • Exploiting Cross-lingual BERT: Further investigation into how cross-lingual BERT abilities can be leveraged for low-resource code-switching setups is recommended.
  • Language-Specific Techniques: Developing and improving language-specific techniques for processing low-resource languages is also highlighted as a future direction.

Summary updated

BibTeX
@inproceedings{aguerotorales2021logistical,
  author = {Marvin Agüero-Torales and David Vilares and Antonio López-Herrera},
  title = {{On the logistical difficulties and findings of Jopara Sentiment Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure