TL;DR
This paper tackles the challenge of sentiment analysis for morphologically rich languages (MRLs) like Hebrew, which present unique linguistic complexities. By systematically comparing different input representations (token vs. morpheme) and neural network architectures, they demonstrate that the optimal choice is highly dependent on the architecture, leading to a new state-of-the-art sentiment analyzer for Hebrew.
What Problem Does This Paper Address?
Modern Hebrew is a morphologically rich language (MRL), meaning words can carry a lot of meaning through prefixes, suffixes, and internal structure, often within a single space-delimited token. This contrasts with languages like English, where words are generally more distinct. Existing neural network (NN) approaches to Natural Language Processing (NLP), including sentiment analysis, have largely been developed and tested on English. Applying these methods to MRLs raises a crucial question: do the linguistic properties of MRLs necessitate different representational choices (how we break down and encode text) and architectural designs for NNs to perform effectively? Specifically, the authors aim to understand how representing text at the token level versus the morpheme level, and how encoding these units (as whole strings or character sequences), impacts sentiment analysis performance for Hebrew, a language for which no dedicated sentiment analyzer currently exists.
Methods
The researchers explored two primary dimensions of representation choices:
- Input Granularity: Comparing token-based representations (using raw, space-delimited words) against morpheme-based representations (where words are segmented into their constituent meaningful units).
- Vocabulary Encoding: Investigating whether to encode these units as string-based (treating each token/morpheme as a unique item) or character-based (representing tokens/morphemes as sequences of characters).
They evaluated these choices across five different architectures: a Linear baseline, Multi-Layer Perceptron (MLP), Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Bidirectional LSTM (BiLSTM). The core idea was to see how these representation and architecture combinations interact to affect sentiment classification accuracy.
Datasets
The authors introduced a new benchmark dataset for Hebrew sentiment analysis. This dataset consists of 12,804 user comments scraped from the official Facebook page of Israel's president. The comments were manually annotated for sentiment (positive, negative, neutral) by two researchers, and the data was also morphologically analyzed and disambiguated to create both token-based and morpheme-based versions.
Key Insights and Ideas
- Architecture Matters: The most significant finding is that the optimal representation choice is not universal but is heavily dependent on the neural network architecture used. What works best for one type of network might not work for another.
- Morphemes for RNNs: Recurrent Neural Networks (RNNs), including LSTMs and BiLSTMs, showed a clear preference for morpheme-based representations. This aligns with the hypothesis that explicit morphological information helps RNNs generalize better, especially with the flexible word order and rich morphology of MRLs.
- Tokens for CNNs/MLPs: Conversely, simpler architectures like MLPs and CNNs performed better with token-based representations. The authors suggest this might be because these models, particularly CNNs with their n-gram-like processing, can leverage the richer context within a token that might be lost when breaking it down into morphemes.
- Character-based Encoding Trade-offs: While character-based encoding can handle out-of-vocabulary (OOV) words by breaking them down, it generally resulted in lower accuracy compared to string-based encoding, suggesting that the richness of full tokens/morphemes is still valuable.
- Hebrew Sentiment is Polarized: The dataset analysis revealed that comments on political posts were highly polarized, with a significant majority being either positive or negative, and a smaller portion being neutral.
Key Results
- The paper achieved state-of-the-art accuracy above 89% for Hebrew sentiment analysis using a token-based CNN with string-based vocabulary encoding.
- For RNNs (LSTM/BiLSTM), morpheme-based representations consistently outperformed token-based ones, with the gap being non-negligible.
- MLPs and CNNs showed a slight preference for token-based representations, especially in string-based encoding.
- The morpheme-based representation showed a lower OOV rate (7.624%) compared to the token-based representation (8.865%) on the test set.
- Qualitative error analysis indicated that models generally performed better on positive sentiment than negative. Disagreements between human annotators and model errors often stemmed from mixed sentiments, sarcasm, or nuanced expressions.
Limitations and Future Work
- Task Dependency: The authors acknowledge that their findings, particularly regarding the preference for token-based representations in CNNs, might be task-dependent. They suggest that for tasks like machine translation or parsing, different trends might emerge.
- Mixed Sentiment Complexity: The analysis of errors highlighted that accurately classifying mixed sentiments remains a significant challenge, often requiring more sophisticated, aspect-based modeling.
- Future Directions: The paper's contribution of a new benchmark and baseline models encourages further research into the interplay between linguistic properties, representation choices, and NN architectures for MRLs. Future work could explore more advanced morphological analysis techniques or aspect-based sentiment analysis.
Summary updated