Will_go at SemEval-2020 Task 9: An Accurate Approach for Sentiment Analysis on Hindi-English Tweets Based on Bert and Pesudo Label Strategy

Wei Bao, Weilong Chen, Wei Bai, Yan Zhuang, Mingyuan Cheng, Xiangyu MaSemEval @ COLING 2020 · DOI

TL;DR

This paper tackles the challenge of sentiment analysis in code-mixed Hindi-English social media text. The authors propose an ensemble model combining a BERT-based approach with pseudo-labeling and a TF-IDF based classifier, achieving a competitive F1 score on the SemEval-2020 Task 9 leaderboard. This work highlights the effectiveness of leveraging pre-trained language models and semi-supervised techniques for nuanced sentiment detection in multilingual online communication.

What Problem Does This Paper Address?

The increasing prevalence of multilingualism in online spaces, particularly on social media, presents a significant challenge for sentiment analysis. Users frequently mix languages within a single post (code-mixing), which can obscure the intended sentiment. This is especially common in societies like India, where Hindi and English are often interwoven. Traditional sentiment analysis models, trained on monolingual data, struggle to accurately interpret these mixed-language texts. The paper aims to develop a robust sentiment analysis system specifically for Hindi-English code-mixed tweets, a task that requires understanding the nuances of both languages and their interplay.

Methods

The core of the proposed solution is an ensemble model that combines two main components:

  • BERT Model with Multi-Sample Dropout and Pseudo-Label Strategy: The authors leverage the power of BERT, a pre-trained transformer model, for its rich semantic understanding. They found that a standard English BERT model performed better than a multilingual BERT on this specific task, likely due to the significant proportion of English words in the dataset. To enhance its performance and robustness, they incorporated:
  • Multi-Sample Dropout: This technique, applied during training, involves creating multiple dropout samples and averaging their losses. This helps prevent overfitting and improves generalization.
  • Pseudo-Labeling: This semi-supervised learning method expands the training data. After initial training on labeled data, the model predicts labels for unlabeled test data. These predicted "pseudo-labels" are then added back to the training set, allowing the model to learn from a larger, albeit partially noisy, dataset.
  • TF-IDF based SGDClassifier: Complementing the BERT model, they use a TF-IDF (Term Frequency-Inverse Document Frequency) approach with an SGDClassifier. TF-IDF is a classic technique for weighting words based on their importance in a document and across a corpus. They combined word n-grams (1-3) and character n-grams (4-6) to capture both lexical and sub-lexical features.

The final ensemble model is a weighted blend of the BERT model with pseudo-labeling and the TF-IDF based SGDClassifier.

Datasets

The paper evaluates its approach on the dataset for SemEval-2020 Task 9, which consists of Hindi-English code-mixed tweets. The specific details of this dataset, such as its size and distribution of sentiment classes, are not explicitly detailed beyond its use in the competition.

Key Insights and Ideas

  • English BERT Outperforms Multilingual BERT: Surprisingly, a standard English BERT model achieved better results than a multilingual BERT on this Hindi-English code-mixed dataset. This suggests that the sheer volume of English words in the dataset made the English-centric model more effective, even when dealing with code-mixing.
  • Pseudo-Labeling Significantly Boosts Performance: The pseudo-labeling strategy proved crucial in improving the BERT model's accuracy. This highlights the value of semi-supervised learning when labeled data is scarce or expensive to obtain, especially in specialized domains like code-mixed text.
  • Ensemble of Diverse Models is Key: Combining the deep semantic understanding of BERT with the statistical word frequency insights from TF-IDF led to the best overall performance. This demonstrates that a hybrid approach, leveraging different modeling paradigms, can capture a more comprehensive view of sentiment.
  • Character N-grams Add Value: The inclusion of character n-grams in the TF-IDF approach helped in capturing sub-word information, which can be particularly useful for handling misspellings or variations in code-mixed text.

Key Results

The authors' ensemble model achieved an average F1 score of 0.686 on the final leaderboard for the SemEval-2020 Task 9.

Here's a breakdown of the F1 scores for different models on the test set:

Approach Negative-Class F1 Score Neutral-Class F1 Score Positive-Class F1 Score Leaderboard F1 Score
Multilingual BERT 0.690 0.636 0.734 0.685
SGDClassifier with TFIDF 0.691 0.634 0.737 0.686
BERT 0.696 0.668 0.783 0.715
BERT with pseudo label 0.738 0.658 0.787 0.725
Ensemble 0.746 0.667 0.789 0.731

The "Ensemble" model, which blends BERT with pseudo-label and TFIDF, achieved the highest Leaderboard F1 Score of 0.731.

Limitations and Future Work

The authors acknowledge that there are still deficiencies in model optimization and parameter setting. They suggest future work could involve:

  • Processing documents mixed in more languages: Extending the approach to handle code-mixing involving more than just Hindi and English.
  • Optimizing and extending the method: Further refining the model's parameters and exploring new techniques to extract richer features across various languages.

Summary updated

BibTeX
@inproceedings{bao2020will,
  author = {Wei Bao and Weilong Chen and Wei Bai and Yan Zhuang and Mingyuan Cheng and Xiangyu Ma},
  title = {{Will\_go at SemEval-2020 Task 9: An Accurate Approach for Sentiment Analysis on Hindi-English Tweets Based on Bert and Pesudo Label Strategy}},
  booktitle = {SemEval @ COLING},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.182},
}
Figure