Hierarchical Interaction Networks with Rethinking Mechanism for Document-Level Sentiment Analysis

Lingwei Wei, Dou Hu, Wei Zhou, Xuehai Tang, Xiaodan Zhang, Xin Wang, Jizhong Han, Songlin Hu

TL;DR

Researchers have developed a new model called HIN-SR that significantly improves document-level sentiment analysis by treating a document's summary as a "subject-oriented" guide. By modeling bidirectional interactions between the summary and the full text and implementing a "rethinking" feedback loop, the model effectively filters out noise and focuses on the most sentiment-rich parts of long, complex documents.

What Problem Does This Paper Address?

Document-level Sentiment Analysis (DSA) is notoriously difficult because long texts often contain "vague semantic links" and a mix of conflicting sentiments. Traditional models often get lost in the weeds, attending to irrelevant words while missing the core subject.

While some recent works have tried using text summarization to help, they often treat the summary and the document as separate entities or fail to let them "talk" to each other during the encoding process. The authors argue that a summary is a goldmine of information—it identifies the main subject (e.g., "Customer Service") which acts as an anchor for understanding which sentiment words (like "bad" or "minimal") actually matter.

Methods

The authors propose two major innovations:

  1. Hierarchical Interaction Network (HIN): This architecture captures interactions at three levels:
  2. Character-level: Uses BERT to capture fine-grained semantic features between the summary and document segments.
  3. Segment-level: Employs a Bidirectional Gated Recurrent Unit (Bi-GRU) to model how the summary relates to different chunks of the document.
  4. Document-level: Uses an attention mechanism where the summary acts as a "query" to determine which segments are most important for the final sentiment score.
  5. Sentiment-based Rethinking mechanism (SR): Most neural networks are "feedforward," meaning data flows in one direction. SR introduces a feedback loop. During training, the model uses the actual sentiment label to "rethink" its internal representations. It calculates a "reward" based on how certain it is about the correct label and uses that to reweight low-level features, effectively telling the model: "Focus more on the features that led to this correct conclusion and ignore the noisy ones that caused confusion."

Datasets

The model was evaluated on three public benchmark datasets:

  • Toys & Games (Amazon SNAP): Product reviews with ratings from 1 to 5.
  • Sports & Outdoors (Amazon SNAP): Similar product reviews with 5 classes.
  • Online News Dataset: News articles categorized into 3 sentiment polarities (Positive, Neutral, Negative), where the article title serves as the summary.

Key Insights and Ideas

  • Summaries as Anchors: The core "aha moment" is that a summary isn't just a shorter version of the text; it's a filter. By forcing the document representation to be "subject-oriented" based on the summary, the model ignores irrelevant sentiment (e.g., a reviewer mentioning a "nice" weather day in a "bad" product review).
  • The Power of Feedback: The Rethinking mechanism mimics human cognition. When we realize we've made a mistake (or are unsure), we go back and look at the evidence differently. By propagating the sentiment label back into the weights, the model learns to suppress "confusable" features.
  • Robustness to Length: Unlike many models that degrade sharply as documents get longer, HIN-SR remains relatively stable, suggesting that the summary-guided attention successfully distills long-form content.

Key Results

HIN-SR outperformed various state-of-the-art baselines, including standard BERT and other hierarchical models like HAN:

  • Accuracy: On the News dataset, HIN-SR achieved 89.3% accuracy, outperforming the standard BERT baseline (85.7%).
  • Amazon Reviews: On Toys & Games, it reached 78.1% accuracy, a significant jump over traditional hierarchical models like HAN (69.1%).
  • Ablation Success: Removing the "Summary" or the "Interactions" led to a noticeable drop in performance, proving that the bidirectional relationship between the two is the "secret sauce" of the model.

Limitations and Future Work

  • Fixed Segmentation: In the current implementation, the authors empirically compressed documents into exactly 3 segments ($T=3$). This might be too rigid for extremely varied document lengths.
  • Summary Dependency: The model relies on having a user-generated summary or title. While common in reviews and news, this isn't always available for other types of documents (like legal or medical texts).
  • Future Directions: The authors suggest exploring "inter-document" interactions (how different documents relate to each other) and providing a deeper theoretical analysis of why the rethinking mechanism works so effectively.

Summary updated

Abstract

Document-level Sentiment Analysis (DSA) is more challenging due to vague semantic links and complicate sentiment information. Recent works have been devoted to leveraging text summarization and have achieved promising results. However, these summarization-based methods did not take full advantage of the summary including ignoring the inherent interactions between the summary and document. As a result, they limited the representation to express major points in the document, which is highly indicative of the key sentiment. In this paper, we study how to effectively generate a discriminative representation with explicit subject patterns and sentiment contexts for DSA. A Hierarchical Interaction Networks (HIN) is proposed to explore bidirectional interactions between the summary and document at multiple granularities and learn subject-oriented document representations for sentiment classification. Furthermore, we design a Sentiment-based Rethinking mechanism (SR) by refining the HIN with sentiment label information to learn a more sentiment-aware document representation. We extensively evaluate our proposed models on three public datasets. The experimental results consistently demonstrate the effectiveness of our proposed models and show that HIN-SR outperforms various state-of-the-art methods.
BibTeX
@inproceedings{wei2020hierarchical,
  author = {Lingwei Wei and Dou Hu and Wei Zhou and Xuehai Tang and Xiaodan Zhang and Xin Wang and Jizhong Han and Songlin Hu},
  title = {{Hierarchical Interaction Networks with Rethinking Mechanism for Document-Level Sentiment Analysis}},
  booktitle = {ECML/PKDD},
  year = {2020},
  url = {https://openreview.net/forum?id=iZCNflsSCU},
}

Loading related content...

Figure