TL;DR
This paper tackles the challenge of sentiment analysis for Jopara, a code-switching language blending Guarani and Spanish. The authors introduce the first Jopara sentiment analysis dataset, highlighting the difficulties in collecting low-resource data, and compare traditional and deep learning models, finding transformer models perform best despite not being pre-trained on Guarani.
What Problem Does This Paper Address?
Natural Language Processing (NLP) applications often overlook indigenous languages, leading to a scarcity of resources. This is particularly true in Latin America, where languages like Guarani coexist with dominant ones like Spanish. Jopara, a code-switching language that mixes Guarani and Spanish (with occasional Portuguese and English influences), is a prime example of such a low-resource scenario. While sentiment analysis (SA) has seen significant progress in high-resource languages, applying it to Jopara presents unique challenges due to the limited availability of annotated data and the complexities of code-switching. This paper aims to bridge this gap by developing a Jopara sentiment analysis dataset and evaluating various machine learning and deep learning models on this challenging task.
Methods
The authors explored two main approaches for collecting Jopara tweets:
- Keyword-based Scraping: This initial attempt involved using a list of Guarani keywords to crawl Twitter. However, this method yielded a very small number of relevant tweets, demonstrating the difficulty of relying solely on keywords for low-resource languages.
- Account-based Scraping: A more successful approach involved identifying and scraping tweets from Twitter accounts that predominantly tweet in Guarani. This method, combined with careful filtering and annotation, proved more effective in building a usable dataset.
For sentiment analysis, they compared several models:
- Traditional Machine Learning: Naïve Bayes (specifically Complement Naïve Bayes to address assumption issues) and Support Vector Machines (SVMs) were used as baselines. These models were trained using TF-IDF weighted unigram and bigram representations.
- Deep Learning Models:
- Recurrent Neural Networks (RNNs): Long Short-Term Memory (LSTM) networks were employed, processing tweets as sequences of token vectors.
- Convolutional Neural Networks (CNNs): CNNs were also used for sentiment analysis, a technique known for its effectiveness in text classification.
- Transformer-based Models: Pre-trained language models like BERT (specifically Spanish BERT, "beto-base-uncased") and multilingual BERT (mBERT) were fine-tuned on the Jopara dataset. These models leverage wordpiece tokenization and contextual embeddings.
Datasets
The primary dataset introduced in this paper is JOSA (Jopara Sentiment Analysis dataset).
- Collection: JOSA was created by scraping tweets from user accounts that primarily tweet in Guarani.
- Size: The initial collection yielded over 23,000 tweets, which were then filtered for similarity and language identification, resulting in 3,948 tweets.
- Annotation: The dataset was annotated by two native Guarani and Spanish speakers for language (Guarani, Jopara, or other) and sentiment (positive, neutral, negative).
- Final Size: After filtering for Guarani-dominant tweets, the final dataset comprises 3,491 tweets.
- Versions: Both unbalanced and balanced versions of JOSA were created for experiments. The unbalanced version reflects real-world distribution, while the balanced version aims to mitigate class imbalance.
- Splits: For both versions, the data was split into training (50%), development (10%), and test (40%) sets.
Key Insights and Ideas
- Data Scarcity is a Major Hurdle: The paper strongly emphasizes the significant logistical difficulties in collecting quality data for low-resource languages like Jopara, even for seemingly straightforward tasks like sentiment analysis. Simple keyword-based approaches are often insufficient.
- Code-Switching Complexity: Jopara's mixed nature, with Guarani and Spanish words, presents challenges for language identification and sentiment analysis. Words with identical forms in both languages and loanwords further complicate the task.
- Transformer Models Shine: Despite not being pre-trained on Guarani, transformer-based models (specifically Spanish BERT) achieved the best performance. This suggests their inherent cross-lingual capabilities are highly beneficial, even when the target language is not explicitly included in the pre-training data.
- Traditional Models Remain Competitive: Surprisingly, traditional machine learning models like Naïve Bayes and SVMs performed closely to deep learning models. This is attributed to the low-resource nature of the task, where the limited data might not fully leverage the power of complex deep learning architectures.
- Complementarity of Models: The error analysis revealed that different models exhibit distinct tendencies in misclassifications. This suggests that ensemble methods, combining the strengths of various models, could be a promising avenue for improving performance in low-resource settings.
Key Results
- Dataset Size: The JOSA dataset contains 3,491 tweets for sentiment analysis.
- Model Performance:
- Best Performer: Spanish BERT ("beto-base-uncased") achieved the highest accuracy on both the unbalanced (0.64) and balanced (0.64) JOSA datasets.
- Competitive Baselines: Traditional models like SVM achieved accuracies of 0.55 (unbalanced) and 0.54 (balanced).
- Deep Learning (non-transformer): The best performing BiLSTM-based model achieved 0.49 (unbalanced) and 0.53 (balanced).
- Neutral Class Dominance: The dataset exhibits a tendency towards neutral tweets, likely due to the nature of content from the scraped accounts (news, announcements, etc.).
- Model Tendencies: CNB tended to over-classify tweets as negative, while deep learning models were better at identifying neutral tweets. BETO showed the best performance in identifying positive tweets.
Limitations and Future Work
- Limited Data Size: The JOSA dataset, while the first of its kind, is still relatively small, which impacts the performance of complex models.
- Topic Diversity: The scraped accounts limited the diversity of topics covered in the dataset.
- Neutral Tweet Bias: The dataset has a notable imbalance towards neutral tweets.
- Future Work:
- Ensemble Methods: The authors suggest exploring ensemble methods to combine complementary models for improved performance.
- Exploiting Cross-lingual BERT: Further investigation into how cross-lingual BERT abilities can be leveraged for low-resource code-switching setups is recommended.
- Language-Specific Techniques: Developing and improving language-specific techniques for processing low-resource languages is also highlighted as a future direction.
Summary updated