Hierarchical Interaction Networks with Rethinking Mechanism for Document-level Sentiment Analysis

Lingwei Wei, Dou Hu, Wei Zhou, Xuehai Tang, Xiaodan Zhang, Xin Wang, Jizhong Han, Songlin Hu

Abstract

Document-level Sentiment Analysis (DSA) is more challenging due to vague semantic links and complicate sentiment information. Recent works have been devoted to leveraging text summarization and have achieved promising results. However, these summarization-based methods did not take full advantage of the summary including ignoring the inherent interactions between the summary and document. As a result, they limited the representation to express major points in the document, which is highly indicative of the key sentiment. In this paper, we study how to effectively generate a discriminative representation with explicit subject patterns and sentiment contexts for DSA. A Hierarchical Interaction Networks (HIN) is proposed to explore bidirectional interactions between the summary and document at multiple granularities and learn subject-oriented document representations for sentiment classification. Furthermore, we design a Sentiment-based Rethinking mechanism (SR) by refining the HIN with sentiment label information to learn a more sentiment-aware document representation. We extensively evaluate our proposed models on three public datasets. The experimental results consistently demonstrate the effectiveness of our proposed models and show that HIN-SR outperforms various state-of-the-art methods.

Introduction

Document-level Sentiment Analysis (DSA) presents unique challenges compared to sentence-level tasks. Long texts often contain "vague semantic links" and a complex mixture of conflicting sentiments. Traditional models frequently struggle with noise, attending to irrelevant words while missing the core subject. While recent approaches have leveraged text summarization to guide sentiment extraction, they often treat the summary and the document as separate entities, failing to model the bidirectional interactions between them during encoding.

The authors argue that a summary serves as a "subject-oriented" guide. It identifies the main subject (e.g., "Customer Service"), which acts as an anchor for understanding which sentiment-bearing words (e.g., "bad" or "minimal") are truly relevant to the overall document sentiment.

Proposed Methodology: HIN-SR

The researchers developed HIN-SR, a model that combines Hierarchical Interaction Networks (HIN) with a Sentiment-based Rethinking (SR) mechanism.

Architecture of the Hierarchical Interaction Network (HIN) with the Sentiment-based Rethinking (SR) mechanism.

Hierarchical Interaction Network (HIN)

The HIN architecture captures interactions at three distinct granularities:

  • Character-level Interaction: Utilizes a BERT layer to capture fine-grained semantic features between the summary and document segments.
  • Segment-level Interaction: Employs a Bidirectional Gated Recurrent Unit (Bi-GRU) to model the contextual relationship between the summary and different document chunks.
  • Document-level Interaction: Uses an attention mechanism where the summary acts as a "query" to determine which segments are most critical for the final sentiment classification.

Sentiment-based Rethinking (SR) Mechanism

Unlike standard feedforward neural networks, the SR mechanism introduces a feedback loop. During training, the model uses the actual sentiment label to "rethink" its internal representations. It calculates a "reward" based on its confidence in the correct label and uses this signal to reweight low-level features. This process effectively instructs the model to focus on features that lead to correct conclusions while suppressing noisy or confusing information.

Experimental Evaluation

The model was evaluated on three public benchmark datasets:

  • Amazon SNAP (Toys & Games / Sports & Outdoors): Product reviews with ratings from 1 to 5.
  • Online News Dataset: News articles categorized into three sentiment polarities (Positive, Neutral, Negative), using article titles as summaries.

Key Results

HIN-SR consistently outperformed state-of-the-art baselines, including standard BERT and Hierarchical Attention Networks (HAN):

  • News Dataset: HIN-SR achieved 89.3% accuracy, compared to 85.7% for the standard BERT baseline.
  • Amazon Reviews (Toys & Games): The model reached 78.1% accuracy, a significant improvement over HAN (69.1%).
  • Ablation Studies: Removing either the summary or the interaction components led to a noticeable performance drop, confirming that the bidirectional relationship between summary and document is essential.

Discussion and Insights

The core innovation lies in treating summaries as anchors. By forcing document representations to be "subject-oriented," the model successfully filters out irrelevant sentiment (e.g., a reviewer mentioning "nice weather" in an otherwise negative product review). Furthermore, the Rethinking mechanism mimics human cognition; when unsure, the model "goes back" to re-evaluate the evidence, learning to suppress "confusable" features over time.

While effective, the model currently uses a fixed segmentation strategy (e.g., documents are compressed into $T=3$ segments), which may be too rigid for documents of varying lengths. Additionally, it relies on the availability of user-generated summaries or titles. Future work may explore automated summary generation for documents lacking them and investigate inter-document interactions.

Conclusion

HIN-SR represents a significant step forward in DSA by effectively utilizing summaries to guide document representation. The combination of multi-level bidirectional interactions and a feedback-driven rethinking mechanism allows the model to distill sentiment-rich information from complex, noisy texts more effectively than previous architectures.

Summary updated

Abstract

Document-level Sentiment Analysis (DSA) is more challenging due to vague semantic links and complicate sentiment information. Recent works have been devoted to leveraging text summarization and have achieved promising results. However, these summarization-based methods did not take full advantage of the summary including ignoring the inherent interactions between the summary and document. As a result, they limited the representation to express major points in the document, which is highly indicative of the key sentiment. In this paper, we study how to effectively generate a discriminative representation with explicit subject patterns and sentiment contexts for DSA. A Hierarchical Interaction Networks (HIN) is proposed to explore bidirectional interactions between the summary and document at multiple granularities and learn subject-oriented document representations for sentiment classification. Furthermore, we design a Sentiment-based Rethinking mechanism (SR) by refining the HIN with sentiment label information to learn a more sentiment-aware document representation. We extensively evaluate our proposed models on three public datasets. The experimental results consistently demonstrate the effectiveness of our proposed models and show that HIN-SR outperforms various state-of-the-art methods.
BibTeX
@inproceedings{wei2020hierarchical,
  author = {Lingwei Wei and Dou Hu and Wei Zhou and Xuehai Tang and Xiaodan Zhang and Xin Wang and Jizhong Han and Songlin Hu},
  title = {{Hierarchical Interaction Networks with Rethinking Mechanism for Document-level Sentiment Analysis}},
  booktitle = {ECML/PKDD},
  year = {2020},
  url = {https://openreview.net/forum?id=iZCNflsSCU},
}
Figure