Subsentential Sentiment on a Shoestring: A Crosslingual Analysis of Compositional Classification

Michael Haas, Yannick VersleyNAACL 2015

TL;DR

This paper tackles the challenge of performing fine-grained sentiment analysis on individual phrases within sentences, especially in low-resource languages like German. They introduce the Heidelberg Sentiment Treebank (HeiST) and explore various methods, including cross-lingual projection and lexicon-based approaches, to achieve this, finding that complex models like RNTNs struggle in data-scarce settings compared to simpler methods.

What Problem Does This Paper Address?

Traditional sentiment analysis often focuses on the overall sentiment of a document or sentence. However, understanding the sentiment of individual phrases (subsentential sentiment) provides a more nuanced view, revealing how specific words and their combinations contribute to the overall feeling. This is particularly useful for tasks like summarization or identifying the precise targets of sentiment. The challenge arises when trying to build such detailed sentiment analysis systems for languages other than English, where annotated data is scarce. Creating these "sentiment treebanks," which annotate sentiment at the phrase level within a syntactic structure, is labor-intensive. This paper aims to investigate how to achieve good performance for compositional sentiment classification in German with limited in-domain, in-language resources, by leveraging cross-lingual methods and existing lexical resources.

Methods

The authors explore several approaches to compositional sentiment classification for German:

  • Supervised Baselines: They train a Recursive Neural Tensor Network (RNTN) model, a state-of-the-art approach for compositional sentiment analysis, on their newly created German dataset. They also use Support Vector Machines (SVMs) with unigram features as a simpler baseline.
  • Cross-lingual Projection: This involves translating German sentences into English, applying an English sentiment analysis model, and then projecting the sentiment labels back to the German syntactic structure. They use machine translation (Google Translate) and word alignment tools for this.
  • Lexicon-based Approaches: They utilize existing German sentiment lexicons (e.g., SentiWS, GermanPolarityClues) and apply two main strategies:
  • Vote-only: The sentiment of a phrase is determined by the sum of the scores of its constituent words.
  • Vote-and-flip: Similar to vote-only, but sentiment values are inverted if a "shifter" word (like a negator) is present.
  • Cluster-based Approaches: They investigate replacing words with their cluster representations, drawing inspiration from work in syntactic parsing that uses clusters to handle data sparsity. They also explore splitting clusters based on sentiment lexicon information.

Datasets

  • Heidelberg Sentiment Treebank (HeiST): This is a newly introduced dataset. It's a German-language counterpart to the Stanford Sentiment Treebank (SSTb), containing opinionated sentences from movie reviews with phrase-level sentiment annotations. It was created by crowdsourcing annotations on movie review summaries, with phrases parsed using the Berkeley Parser and then binarized.
  • Stanford Sentiment Treebank (SSTb): Used for comparison and for training English-language models for cross-lingual experiments.

Key Insights and Ideas

  • RNTNs struggle in data-scarce settings: Surprisingly, the powerful RNTN model, which performs well on larger datasets like SSTb, is significantly impeded by the data sparsity in the HeiST dataset. This suggests that while RNTNs are good at capturing complex compositional interactions, they require substantial data to learn effectively.
  • Simpler methods can be competitive: Lexicon-based approaches and even SVMs with basic features often outperform the RNTN on the HeiST dataset. This highlights the effectiveness of simpler, more data-efficient methods when resources are limited.
  • Cross-lingual projection offers a viable alternative: Translating to English and applying existing models provides a reasonable level of accuracy, demonstrating the utility of cross-lingual transfer for low-resource sentiment analysis, even if it incurs some accuracy loss.
  • Multi-word expressions (MWEs) are a major challenge: The analysis of production rules reveals that MWEs constitute the vast majority of compositional structures in both SSTb and HeiST, and they are particularly difficult for models to handle, posing a significant challenge for compositional sentiment analysis.
  • Noise in crowdsourced data: The authors acknowledge that noise from the crowdsourcing process can slightly distort counts and affect model performance, particularly for MWEs.

Key Results

  • SVMs and lexicon-based methods outperform RNTN: On the HeiST dataset, SVMs with unigram features achieved node accuracies around 85%, and lexicon-based approaches reached accuracies in the low 80s, while the RNTN achieved around 77.6% node accuracy.
  • Cross-lingual projection yields competitive results: The cross-lingual projection approach achieved a node accuracy of approximately 82.3%, which is competitive with supervised methods trained on limited data and better than the RNTN on HeiST.
  • Lexicon-based approaches are robust: Lexicon-based methods, particularly those incorporating negation handling (vote-and-flip), showed consistent performance across different lexicons.
  • MWEs are the hardest: MWEs were the most challenging production type for all tested models, with significantly lower accuracy compared to other types like AVG (averaging).

Limitations and Future Work

  • Data Sparsity: The primary limitation is the data sparsity of the HeiST dataset, which hinders the performance of complex models like RNTNs.
  • Noise in Crowdsourcing: The authors acknowledge that crowdsourced annotations can contain noise, affecting the reliability of the dataset and potentially distorting results.
  • Future Work: The paper suggests further investigation into the nature of MWEs and their impact on compositionality. Exploring more sophisticated ways to leverage cross-lingual information and improving lexicon-based methods, especially for handling negation and intensifiers, are also promising directions. The authors also hint at the potential of latent-variable models for syntax to deal with data sparsity, suggesting their application to sentiment analysis could be fruitful.

Summary updated

BibTeX
@inproceedings{haas2015subsentential,
  author = {Michael Haas and Yannick Versley},
  title = {{Subsentential Sentiment on a Shoestring: A Crosslingual Analysis of Compositional Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure