Tourism Review Sentiment Classification Using a Bidirectional Recurrent Neural Network with an Attention Mechanism and Topic-Enriched Word Vectors

Qin Li, Shaobo Li, Jie Hu, Sen Zhang, Jianjun HuSustainability 2018 · DOI

TL;DR

This paper introduces BiGRULA, a novel sentiment classification model for tourism reviews that combines topic modeling (lda2vec) with a bidirectional recurrent neural network (BiGRU) and an attention mechanism. By enriching word embeddings with topic information and allowing the model to focus on important words, BiGRULA achieves state-of-the-art performance on benchmark datasets and demonstrates its effectiveness in understanding nuanced customer opinions in hotel reviews.

What Problem Does This Paper Address?

The explosion of online reviews for tourism services presents a rich source of information about customer sentiment. However, accurately analyzing these reviews is challenging. Existing sentiment analysis methods often struggle because they either:

  • Ignore the underlying topics: They treat all words equally, failing to capture how sentiment is tied to specific aspects of a service (e.g., "cleanliness" vs. "location").
  • Don't account for word importance: Not all words contribute equally to the overall sentiment. Some words are crucial indicators, while others are less significant.

This paper aims to address these limitations by developing a model that can simultaneously understand the topics discussed in reviews and dynamically weigh the importance of individual words for sentiment classification.

Methods

The core of the proposed BiGRULA model is a synergistic combination of three key components:

  • lda2vec: This is a novel topic modeling technique that merges Latent Dirichlet Allocation (LDA) with word embeddings (like word2vec). It goes beyond traditional LDA by not only identifying topics but also enriching word vectors with topic-specific context. This means a word like "bank" could have different vector representations depending on whether the topic is "finance" or "riverbank." This provides a more nuanced understanding of word meaning within a document.
  • Bidirectional Gated Recurrent Unit (BiGRU): This is a type of recurrent neural network (RNN) that processes sequences of data (like words in a sentence) in both forward and backward directions. GRUs are designed to handle long-term dependencies in sequential data, mitigating the vanishing gradient problem common in older RNNs. This allows the model to capture the flow and context of sentences effectively.
  • Attention Mechanism: This mechanism allows the BiGRU model to dynamically focus on the most relevant words in a review when making a sentiment prediction. Instead of treating all words equally, the attention mechanism assigns different weights to words based on their importance to the overall sentiment. This is crucial for capturing subtle nuances and specific aspects driving customer opinion.

The BiGRULA model first uses lda2vec to generate topic-enhanced word vectors. These vectors are then fed into the BiGRU network, which uses the attention mechanism to weigh their importance before making the final sentiment classification.

Datasets

The paper evaluates its model on several datasets:

  • 20 Newsgroup: A classic multi-class text classification dataset used to evaluate the performance of the lda2vec topic-enhanced word embeddings.
  • IMDB Movie Review Dataset: A widely used binary sentiment classification dataset (positive/negative reviews) for evaluating the overall performance of the BiGRULA model against other neural network architectures.
  • ChnSentiCorp-Htl-unba-10000 Hotel Review Dataset: A Chinese hotel review dataset used to demonstrate the practical application of BiGRULA in a real-world tourism scenario, focusing on extracting coherent topics and performing sentiment analysis.

Key Insights and Ideas

  • Topic-Enhanced Embeddings Matter: The research highlights that enriching word embeddings with topic information (via lda2vec) significantly improves performance compared to standard word embeddings like Skip-Gram or GloVe, especially in capturing richer document representations.
  • Attention is Crucial for Nuance: The attention mechanism is not just a performance booster; it's essential for understanding why a review is positive or negative by allowing the model to focus on the most impactful words.
  • Coherent Topics for Better Understanding: Applying lda2vec to hotel reviews yielded more interpretable and human-understandable topics compared to traditional LDA, suggesting its suitability for shorter, more specific texts.
  • Bidirectional Context is Key: The use of BiGRU effectively captures the sequential nature of language and contextual dependencies, which is vital for accurate sentiment interpretation.

Key Results

  • Outperforming Baselines: On the 20 Newsgroup dataset, lda2vec achieved the highest accuracy (80.2%) compared to BOW, LDA, Skip-Gram, PV, and GloVe.
  • State-of-the-Art on IMDB: The BiGRULA model achieved a test accuracy of 0.894 on the IMDB dataset, outperforming other neural network models like CNN, LSTM, and CNN+LSTM, as well as various machine learning approaches.
  • Effective Hotel Review Analysis: When applied to Chinese hotel reviews, BiGRULA achieved a high binary sentiment classification accuracy of 93.1% after just four training epochs. The t-SNE visualizations showed a clear separation between positive and negative review embeddings, demonstrating the model's ability to differentiate emotions.

Limitations and Future Work

  • Word-Level Attention: The authors acknowledge that BiGRULA currently uses a word-level attention mechanism. They suggest that incorporating a hierarchical attention network (HAN) with sentence-level attention could further enhance the model's training ability and representation power.
  • Generalizability: While effective on tourism and movie review datasets, further testing on diverse domains would be beneficial.
  • Topic Coherence in Short Texts: While lda2vec showed improvement over LDA for hotel reviews, extracting highly coherent topics from very short texts remains an area for potential improvement.

The authors propose future work focusing on integrating hierarchical attention and exploring the model's application in hotel recommendation systems and for analyzing consumer satisfaction strategies.

Summary updated

Abstract

Sentiment analysis of online tourist reviews is playing an increasingly important role in tourism. Accurately capturing the attitudes of tourists regarding different aspects of the scenic sites or the overall polarity of their online reviews is key to tourism analysis and application. However, the performances of current document sentiment analysis methods are not satisfactory as they either neglect the topics of the document or do not consider that not all words contribute equally to the meaning of the text. In this work, we propose a bidirectional gated recurrent unit neural network model (BiGRULA) for sentiment analysis by combining a topic model (lda2vec) and an attention mechanism. Lda2vec is used to discover all the main topics of review corpus, which are then used to enrich the word vector representation of words with context. The attention mechanism is used to learn to attribute different weights of the words to the overall meaning of the text. Experiments over 20 NewsGroup and IMDB datasets demonstrate the effectiveness of our model. Furthermore, we applied our model to hotel review data analysis, which allows us to get more coherent topics from these reviews and achieve good performance in sentiment classification.
BibTeX
@inproceedings{li2018tourism,
  author = {Qin Li and Shaobo Li and Jie Hu and Sen Zhang and Jianjun Hu},
  title = {{Tourism Review Sentiment Classification Using a Bidirectional Recurrent Neural Network with an Attention Mechanism and Topic-Enriched Word Vectors}},
  booktitle = {Sustainability},
  year = {2018},
  doi = {10.3390/su10093313},
}
Figure