INSIGHT-1 at SemEval-2016 Task 5: Deep Learning for Multilingual Aspect-based Sentiment Analysis

Sebastian Ruder, Parsa Ghaffari, John G. Breslin

TL;DR

This paper presents a deep learning approach for multilingual aspect-based sentiment analysis, achieving competitive results across various languages and domains. By framing aspect extraction as a multi-label classification problem and using a convolutional neural network (CNN) for both extraction and sentiment analysis, the authors demonstrate the effectiveness of deep learning in this complex NLP task.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, has been a long-standing area of research. However, traditional sentiment analysis often provides a general sentiment for an entire piece of text. In many real-world scenarios, like product reviews, users express opinions about specific aspects of an entity (e.g., the "battery life" of a laptop, or the "service" at a restaurant). Aspect-based sentiment analysis (ABSA) aims to go deeper by identifying these aspects and the sentiment expressed towards them. While much prior work focused on English, SemEval-2016 Task 5 introduced a significant challenge: performing ABSA across multiple languages and domains. This paper tackles this multilingual ABSA problem, aiming to develop a robust and language-independent deep learning solution.

Methods

The core of the INSIGHT-1 system is a deep learning architecture based on Convolutional Neural Networks (CNNs). The authors adapt a CNN structure, commonly used for text classification, for two key subtasks:

  • Aspect Extraction: This is framed as a multi-label classification problem. For a given sentence, the CNN outputs probabilities for various predefined aspects. A threshold is then applied to determine which aspects are present.
  • Aspect-Based Sentiment Analysis: Here, the model concatenates the word embeddings of the input sentence with an "aspect vector." This combined representation is then fed through a CNN to predict the sentiment polarity (positive, negative, neutral) towards that specific aspect.

A key innovation is the handling of multilingualism. For aspect extraction, they treat it as a multi-label problem and use a thresholding mechanism. For sentiment analysis, they concatenate word embeddings with aspect embeddings. Notably, for English, they use a shared embedding space for both words and aspect tokens, exploiting pre-trained embeddings. For other languages, they maintain separate embedding spaces for aspect tokens, as direct translation of aspect terms didn't yield benefits. They also employ techniques like max-over-time pooling for feature extraction and a softmax layer for output probabilities.

Datasets

The paper evaluates its approach on the SemEval-2016 Task 5 dataset, which comprises data for multilingual aspect-based sentiment analysis across various languages and domains, including:

  • Languages: English (EN), Spanish (SP), French (FR), Russian (RU), Dutch (DU), Turkish (TU), Arabic (AR), Chinese (CH).
  • Domains: Restaurants (REST), Laptops (LAPT), Phones (PHNS), Cameras (CAME), Hotels (HOTE).

The authors also mention using pre-trained GloVe word embeddings trained on a large Common Crawl corpus for initialization.

Key Insights and Ideas

  • Deep Learning for Multilingual ABSA is Viable: The paper strongly demonstrates that deep learning, specifically CNNs, can achieve competitive performance in multilingual aspect-based sentiment analysis, even without extensive feature engineering or language-specific models.
  • Framing Aspect Extraction as Multi-Label Classification: This approach allows the model to identify multiple aspects within a single sentence, which is crucial for real-world text.
  • Aspect Embeddings Enhance Sentiment Prediction: By incorporating aspect information directly into the input for sentiment analysis, the model can better understand the target of the sentiment.
  • Language-Specific Embedding Strategies Matter: The decision to use shared embeddings for English (leveraging pre-trained resources) versus separate embeddings for other languages highlights the nuanced approach needed for multilingual NLP.
  • Pre-trained Embeddings Provide a Significant Boost: Initializing models with general-purpose pre-trained embeddings, especially for English, leads to a noticeable performance improvement.

Key Results

The INSIGHT-1 system achieved strong results in SemEval-2016 Task 5:

  • Aspect Category Detection (Slot 1): The system placed first or second in 5 out of 11 language-domain pairs.
  • Sentiment Polarity (Slot 3): The system placed first or second in 7 out of 11 language-domain pairs.
  • Performance Across Languages: While competitive, the performance gap was largest for English, Spanish, French, and Turkish compared to the best-performing systems.
  • Impact of Pre-trained Embeddings: Using pre-trained embeddings provided a significant performance boost, particularly in the English restaurants domain.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Performance Gap in Certain Languages: The system's performance lagged behind the best systems in some languages (English, Spanish, French, Turkish). They hypothesize that initializing with embeddings trained on large monolingual corpora in the target language could improve this.
  • Incorporating Domain Information: Best-performing systems often leverage domain-specific information. The authors suggest pre-training on large in-domain corpora (like the Yelp Dataset Challenge) could further enhance results.
  • Robustness of Multi-Label Classification: The current multi-label enforcement happens after prediction via a threshold, which might discard valid aspects or include erroneous ones. Integrating the multi-label condition more naturally into the architecture (e.g., through error functions) is suggested.
  • Handling Complex Sentiments: For aspect-based sentiment analysis, simply concatenating word and aspect vectors might not be expressive enough to model truly aspect-dependent sentiment, especially in sentences with opposing sentiments towards different aspects. Future work could focus on better modeling of these complex surface forms and their relation to aspect instantiations.
  • Contextual Information: While they experimented with using preceding and subsequent sentences, it did not improve results. Further exploration of context might be beneficial.

Summary updated

Abstract

This paper describes our deep learning-based approach to multilingual aspect-based sentiment analysis as part of SemEval 2016 Task 5. We use a convolutional neural network (CNN) for both aspect extraction and aspect-based sentiment analysis. We cast aspect extraction as a multi-label classification problem, outputting probabilities over aspects parameterized by a threshold. To determine the sentiment towards an aspect, we concatenate an aspect vector with every word embedding and apply a convolution over it. Our constrained system (unconstrained for English) achieves competitive results across all languages and domains, placing first or second in 5 and 7 out of 11 language-domain pairs for aspect category detection (slot 1) and sentiment polarity (slot 3) respectively, thereby demonstrating the viability of a deep learning-based approach for multilingual aspect-based sentiment analysis.
BibTeX
@inproceedings{ruder2016insight,
  author = {Sebastian Ruder and Parsa Ghaffari and John G. Breslin},
  title = {{INSIGHT-1 at SemEval-2016 Task 5: Deep Learning for Multilingual Aspect-based Sentiment Analysis}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  url = {https://openreview.net/forum?id=nQ9PiPhWGM},
}
Figure