CUET-NLP@DravidianLangTech-ACL2022: Exploiting Textual Features to Classify Sentiment of Multimodal Movie Reviews

Nasehatul Mustakim, Nusratul Jannat, Md Hasan, Eftekhar Hossain, Omar Sharif, Mohammed Moshiul HoqueACL 2022

TL;DR

This paper tackles the challenge of sentiment analysis for movie reviews in low-resource Dravidian languages (Tamil and Malayalam) by focusing solely on textual features. They found that a simple Decision Tree model, surprisingly, outperformed more complex deep learning approaches, highlighting the importance of feature engineering and model selection even with limited data.

What Problem Does This Paper Address?

The proliferation of social media has led to a massive increase in user-generated content, making sentiment analysis a crucial area for understanding public opinion. While significant progress has been made in high-resource languages like English, low-resource languages, such as Tamil and Malayalam, lag behind due to a scarcity of annotated datasets and specialized language processing tools. This paper specifically addresses the task of classifying sentiment in multimodal movie reviews (which include video, audio, and text) for these Dravidian languages, but focuses exclusively on the textual modality due to data limitations. The challenge lies in accurately capturing the nuances of sentiment expressed in these languages, especially when dealing with limited training data and the inherent ambiguity of language.

Methods

The authors explored both traditional Machine Learning (ML) and Deep Learning (DL) techniques. For ML models, they employed TF-IDF to extract unigram textual features. The ML classifiers used were Logistic Regression (LR), Decision Tree (DT), Support Vector Machine (SVM), and Naive Bayes (MNB).

For DL models, they utilized pre-trained Word2vec and FastText embeddings to capture richer semantic information. The DL architectures investigated were Bidirectional Long Short-Term Memory (BiLSTM) and a combination of Convolutional Neural Network (CNN) with BiLSTM (CNN+BiLSTM). These models were trained on the extracted text features.

Datasets

The dataset used in this study was provided by the shared task organizers. It consists of multimodal movie reviews (videos, audios, and text) that were manually annotated into five sentiment categories: highly positive (HPS), positive (PS), neutral (NT), negative (NE), and highly negative (HNE). The dataset was split into training, validation, and test sets. The paper specifically focused on the Tamil and Malayalam portions of this dataset.

  • Tamil Data: 64 samples
  • Malayalam Data: 70 samples

The paper notes that the dataset is relatively small, particularly for the "highly positive" and "highly negative" classes, which poses a significant challenge for model training and evaluation.

Key Insights and Ideas

  • Simplicity Wins (Sometimes): The most striking finding is that a traditional Decision Tree (DT) model, using TF-IDF features, outperformed more sophisticated deep learning models like CNN+BiLSTM. This suggests that for this specific dataset and task, careful feature selection and a well-tuned DT can be more effective than complex neural architectures, especially when data is scarce.
  • The Curse of Small Datasets: The paper highlights how extremely imbalanced class distributions (e.g., very few samples for "highly positive" and "highly negative") severely impact model performance, leading to zero f1-scores for those classes. This is a common problem in low-resource NLP.
  • Textual Features Alone are Limited: While the study focused on text, the authors acknowledge that excluding video and audio features likely limited the overall performance, as these modalities often carry significant sentiment cues.

Key Results

The primary metric for evaluating model performance was the macro f1-score.

  • The Decision Tree (DT) model achieved the highest macro f1-score of 0.24, outperforming all other ML and DL models.
  • The DT model showed an 86.6% increase in performance compared to the average of ML models and a 140% improvement over the best DL model (CNN+BiLSTM).
  • Surprisingly, the DL models showed poorer performance compared to their ML counterparts. The CNN+BiLSTM model achieved a macro f1-score of 0.10.
  • The class-wise f1-scores reveal that the "positive" class had the highest f1-score in the DT model (0.80) due to its larger number of instances. The "highly positive" and "highly negative" classes had the lowest f1-scores (0.0) due to their minimal sample sizes.

Limitations and Future Work

  • Limited Data: The most significant limitation acknowledged is the small size and imbalance of the dataset. This directly impacts the ability of models to learn robust representations, especially for minority classes.
  • Text-Only Approach: The study exclusively used textual features, neglecting the valuable information present in video and audio modalities. This is a major reason for the suboptimal performance of even the best models.
  • Future Work: The authors propose several avenues for future research:
  • Multimodal Integration: Incorporating video and audio features alongside text to leverage the full richness of the data.
  • Data Augmentation: Employing data oversampling techniques or utilizing larger open-source corpora to create synthetic data and improve model generalization.
  • Advanced Models: Exploring state-of-the-art transformer-based models like IndicBERT, mBERT, XLM-R, and MuRIL for improved performance.

Summary updated

BibTeX
@inproceedings{mustakim2022cuet,
  author = {Nasehatul Mustakim and Nusratul Jannat and Md Hasan and Eftekhar Hossain and Omar Sharif and Mohammed Moshiul Hoque},
  title = {{CUET-NLP@DravidianLangTech-ACL2022: Exploiting Textual Features to Classify Sentiment of Multimodal Movie Reviews}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure