English-Malay Word Embeddings Alignment for Cross-lingual Emotion Classification with Hierarchical Attention Network

Ying Hao Lim, Jasy Suet Yan LiewACL 2022

TL;DR

This paper tackles the challenge of emotion classification in Malay, a low-resource language, by leveraging English resources through cross-lingual word embeddings and a hierarchical attention network. Their approach achieves competitive performance with significantly reduced computational cost, making cross-lingual sentiment analysis more accessible.

What Problem Does This Paper Address?

Classifying emotions in text is crucial for understanding public opinion, especially during critical events. While this task is well-studied for resource-rich languages like English, it remains a significant challenge for languages with limited annotated data, such as Malay. Traditional methods often rely on machine translation, which can struggle with the nuances of informal text like social media posts. This paper aims to bridge this gap by developing a cross-lingual approach that effectively transfers knowledge from English to Malay for emotion classification, specifically focusing on social media data.

Methods

The core of this work lies in creating aligned English and Malay word embeddings and then using them to train a hierarchical attention network (HAN).

  • Cross-lingual Word Embeddings: They start with pre-trained English (EWE) and Malay (MWE) monolingual word embeddings. To align them, they use a bilingual lexicon (T-BL) and apply an orthogonal transformation based on Singular Value Decomposition (SVD).
  • Embeddings Refinement: To improve the quality of these aligned embeddings, they introduce a novel refinement step. This process pulls words with similar sentiment closer in the embedding space and pushes dissimilar words apart, guided by sentiment keywords derived from English norms (E-ANEW) and WordNet synsets, all without human intervention.
  • Hierarchical Attention Network (HAN): The emotion classification model is a HAN. It processes tweets by first encoding words within sentences and then encoding sentences within the tweet. Crucially, it incorporates a multi-head self-attention mechanism at the sentence level, which they found to be more effective than other attention configurations. This model is pre-trained on English tweets and then fine-tuned on Malay tweets.

Datasets

  • English Training Tweets: A subset of the EmoTweet-28 corpus, filtered for specific emotion categories (anger, fear, happiness, love, none, sadness, surprise) and single-label tweets.
  • Malay Evaluation Tweets: A subset of tweets from the Malaya Documentation corpus (Husein, 2018). This dataset was initially labeled by a rule-based classifier and then manually validated by three native speakers. The authors also filtered out Indonesian Malay tweets.
  • Bilingual Lexicons:
    • Training Bilingual Lexicon (T-BL): Derived from the Malaya Documentation corpus, used for mapping embeddings.
  • Gold Standard Bilingual Lexicon (G-BL): Manually created from T-BL by translating English words and verifying Malay translations using a DBP dictionary.
  • New Bilingual Lexicon (N-BL): Created by translating English words from T-BL using Google Cloud Translation API.

Key Insights and Ideas

  • Word-level attention transfer is feasible: The research demonstrates that attention weights learned on English tweets can be effectively transferred to Malay tweets when using aligned cross-lingual embeddings. This means words carrying similar emotional weight in English will do so in Malay.
  • Refined embeddings are crucial: The proposed unsupervised refinement of cross-lingual word embeddings significantly improves their quality, leading to better mapping and, consequently, better downstream classification performance.
  • Hierarchical attention with sentence-level attention is effective: The HAN architecture, particularly with the multi-head self-attention at the sentence level, proves adept at capturing the hierarchical structure of tweets for emotion classification.
  • Computational efficiency matters: The proposed model achieves competitive results with substantially less computational time and resources compared to large pre-trained multilingual models like mBERT and XLM-R, making it a more practical solution for low-resource languages.

Key Results

  • Zero-shot learning: The HMAN model outperforms mBERT by 2.4% in zero-shot learning. While XLM-R-base achieves the highest score (0.5193), HMAN is significantly more computationally efficient.
  • Few-shot learning: After fine-tuning, HMAN achieves a macro F1-score of 0.8836, outperforming HAN by 7.3% and being on par with mBERT. It lags behind XLM-R-base by only 4.3% while being 6-7 times faster.
  • Embeddings quality: The refined Malay embeddings (F-MWE) showed improved mapping quality (P@10) compared to the original Malay embeddings (MWE) in both T-BL and N-BL evaluations.
  • Fine-tuning layers: Fine-tuning only the output layer resulted in a drastic performance drop, highlighting the importance of fine-tuning the entire model for cross-lingual tasks.

Limitations and Future Work

  • Dataset bias: The authors acknowledge that their created Malay emotion corpus might contain biases as it was derived from an existing, rule-based labeled corpus and not fully annotated from scratch.
  • Informal corpora: The study focuses on informal social media text. Evaluating on standard Malay corpora would provide a more comprehensive understanding of the model's performance.
  • Future work:
    • Explore sentence-level mapping and refinement for embeddings.
  • Expand the Malay emotion corpus with more annotations.
  • Investigate semi-supervised and unsupervised approaches for cross-lingual embeddings.
  • Evaluate the model on standard Malay corpora to assess performance in formal contexts.

Summary updated

BibTeX
@inproceedings{lim2022english,
  author = {Ying Hao Lim and Jasy Suet Yan Liew},
  title = {{English-Malay Word Embeddings Alignment for Cross-lingual Emotion Classification with Hierarchical Attention Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure