Improving Sentiment Classification in Slovak Language

Samuel Pecar, Marian Simko, Maria BielikovaACL 2019

TL;DR

This paper tackles the challenge of sentiment classification in low-resource languages, specifically Slovak, by adapting state-of-the-art neural network architectures and word representations. The authors demonstrate that their proposed models, leveraging deep contextualized word embeddings like ELMo and attention mechanisms, significantly outperform previous benchmarks for Slovak sentiment analysis on both customer reviews and Twitter data.

What Problem Does This Paper Address?

Sentiment analysis, a crucial task in Natural Language Processing (NLP), has seen tremendous progress, largely driven by advancements in neural networks. However, most research and development have focused on high-resource languages like English, leaving many other languages, particularly those with fewer available annotated datasets (low-resource languages), underserved. This gap means that the benefits of cutting-edge NLP techniques are not accessible to a significant portion of the world's population. This paper specifically addresses this issue by investigating methods to improve sentiment classification for Slovak, a Slavic language that, like many others, lacks extensive labeled data. The authors aim to show that existing advanced NLP techniques can be effectively adapted and applied to such languages, potentially unlocking new avenues for sentiment analysis research beyond English.

Methods

The authors propose a general neural network architecture for sentiment classification, comprising four key blocks: word representations, recurrent neural network layers, a self-attention layer, and a linear decoder.

  • Word Representations: They experiment with various word representations, including randomly initialized embeddings (LookUp), and pre-trained embeddings like ELMo (for Slovak), fastText (for Slovak), and word2vec (trained on the Slovak National Corpus). This is a critical component, as the quality of word embeddings significantly impacts downstream tasks.
  • Recurrent Neural Network (RNN) Layers: They utilize LSTM and Bi-LSTM architectures, exploring both single and stacked layers, with a fixed hidden size of 512. Bi-directional LSTMs are generally preferred for capturing context from both past and future words in a sequence.
  • Self-Attention Layer: An attention mechanism is incorporated to allow the model to focus on the most informative words within a sentence, assigning them higher weights when computing the final representation.
  • Model Ensemble: To further boost performance, they experiment with model ensembling, where multiple models are trained and their predictions are combined either through voting or by averaging probabilities. This is a common technique to improve robustness and accuracy.

The paper systematically evaluates different combinations of these architectural components and word representations.

Datasets

The authors evaluate their models on two distinct datasets:

  • Reviews3: This dataset comprises customer reviews for various services, manually labeled by two annotators. It's noted that the agreement between annotators was not very high, leading to the categorization of reviews into three classes: positive, negative, and neutral (which includes slightly positive or negative reviews). The dataset contains 5320 reviews.
  • Twitter: This dataset consists of tweets in Slovak, originally labeled by Mozetič et al. (2016). Due to the unavailability of some original tweets, the evaluation was performed on tweets accessible via the standard Twitter API. This dataset is significantly larger, containing 50710 tweets.

For the Reviews3 dataset, a 5-fold cross-validation was employed due to its smaller size. For the Twitter dataset, a standard train/validation/test split of 8:1:1 was used.

Key Insights and Ideas

  • Deep Contextualized Embeddings Shine: The most significant performance gains were achieved by using deep contextualized word representations like ELMo, highlighting their power in capturing nuanced meaning, even for low-resource languages.
  • Bidirectionality and Attention Matter: Employing Bi-LSTMs and the self-attention mechanism consistently improved performance over simpler unidirectional LSTMs without attention, demonstrating their value in capturing sequential dependencies and salient information.
  • Model Ensembling Boosts Performance: Combining multiple models through ensembling further enhanced the sentiment classification accuracy, especially for the Twitter dataset, showcasing its effectiveness in improving robustness.
  • Transfer Learning Potential: The success of pre-trained ELMo and fastText for Slovak suggests a strong potential for transfer learning from high-resource languages or large unlabeled corpora to improve NLP tasks in low-resource settings.

Key Results

The paper reports significant improvements over previous work on Slovak sentiment classification.

  • Reviews3 Dataset: The best performing models achieved micro F1 scores exceeding 0.81, a substantial improvement over previously published results which were around 0.75 (using SVM) or 0.72 (using a neural network baseline). The authors note that using word2vec and fastText did not yield significant improvements over randomly initialized embeddings for this specific dataset.
  • Twitter Dataset: The best models achieved micro F1 scores around 0.69-0.70. While this is a strong performance for the dataset, it's noted that the performance is lower than on the Reviews3 dataset. Model ensembling further improved these results slightly, reaching a micro F1 of 0.7008.
  • Comparison with Previous Work: For the Reviews3 dataset, their best models outperformed previous SVM and neural network baselines by more than 6 percentage points in micro F1 score. For the Twitter dataset, their best performing model (ELMO-BiLSTM-att) achieved a micro F1 of 0.6978, which is comparable to or better than the reported "TwoPlaneSVMbin" baseline (0.6840*), though a direct comparison is limited due to dataset differences.

Limitations and Future Work

  • Dataset Size and Quality: The authors acknowledge the limited size of the Reviews3 dataset and the potential impact of low annotator agreement on its quality. The Twitter dataset, while larger, might contain more general sentiment that is harder to classify without additional context.
  • Fine-tuning and Hyperparameters: The authors mention that they used standard setups and did not perform extensive fine-tuning or hyperparameter optimization, suggesting that further improvements could be achieved with more rigorous tuning.
  • Cross-lingual Learning: They propose cross-lingual learning as a promising avenue for future work, where knowledge from multiple languages could be leveraged to mitigate the lack of annotated resources in low-resource languages.
  • ELMo Training Data: The performance of ELMo could potentially be further improved by training it on a larger Slovak corpus, as the current pre-trained version might have been trained on a limited dataset.
  • Contextual Information: For the Twitter dataset, they observed that misclassifications often stemmed from tweets that relied on real-world context or external information not present in the text itself. Future work could explore incorporating such external knowledge.
  • Code Availability: The authors have made their code publicly available on GitHub, which is a positive step for reproducibility and further research.

Summary updated

BibTeX
@inproceedings{pecar2019improving,
  author = {Samuel Pecar and Marian Simko and Maria Bielikova},
  title = {{Improving Sentiment Classification in Slovak Language}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure