Modeling Complex Interactions in Long Documents for Aspect-Based Sentiment Analysis

Zehong Yan, Wynne Hsu, Mong-Li Lee, David Bartram-ShawWASSA 2024 · DOI · OpenReview

TL;DR

This paper introduces DART, a novel hierarchical Transformer architecture designed to tackle the challenge of aspect-based sentiment analysis in long documents. By effectively capturing long-range dependencies and aggregating information at multiple granularities, DART significantly outperforms existing methods, even large language models like GPT-4, on complex sentiment classification tasks.

What Problem Does This Paper Address?

The internet is awash with user-generated content, from product reviews to news articles. Understanding the sentiment expressed towards specific aspects within this content is crucial for businesses and researchers alike. However, existing aspect-based sentiment analysis (ABSA) models often struggle with long documents. In lengthy texts, an aspect might be mentioned indirectly, across many sentences, or even in conflicting ways. Traditional models, often trained on shorter texts, fail to capture these distant contextual cues, leading to inaccurate sentiment predictions. This paper aims to bridge this gap by developing a model that can effectively process and understand sentiment in long documents, considering the nuanced interplay of information across numerous sentences.

Methods

The core of this work is the DART (Document-level Aspect-based Representation Transformer) framework. DART employs a hierarchical approach with several key components:

  • Sentence Encoding Block: This initial stage uses a pre-trained language model (like BERT or BigBird) to generate embeddings for each sentence, crucially incorporating the target aspect into the input. This allows the model to start understanding sentences in relation to specific aspects.
  • Global Context Interaction Block: This is where DART tackles long-range dependencies. It uses two Transformer encoders:
  • The first encoder uses self-attention on the [CLS] tokens of each sentence to capture inter-sentence relationships and create contextually enriched sentence representations.
  • The second encoder refines these representations by combining the context-enriched [CLS] tokens with the original sentence token embeddings, allowing each sentence's representation to be influenced by both its local context and the broader document context.
  • Aspect Aggregation Block: This block is vital for synthesizing the information. It uses a two-level attention mechanism:
  • Local Aggregation: It weighs the importance of each token within a sentence based on its relevance to the aspect and the overall sentence context.
  • Global Aggregation: It then performs a global attentive pooling over the sentence-level aggregated representations to create a unified document representation for the specific aspect. This ensures that sentences more relevant to the aspect have a greater influence.
  • Sentiment Classification Block: Finally, a two-layer MLP takes the aggregated document representation and predicts the sentiment polarity (positive, negative, etc.) for the given aspect.

Datasets

The authors evaluated DART on several datasets:

  • BeerAdvocate: A dataset of beer reviews with ratings for aspects like "feel," "look," "smell," and "taste."
  • TripAdvisor: A dataset of hotel reviews with ratings for aspects such as "value," "room," "location," and "cleanliness."
  • SocialNews: A newly curated dataset from news articles related to social issues, focusing on implicit aspects like "crime-justice," "digital-online," "economic issues," "health," "human rights," and "work." This dataset is notable for its inclusion of long documents and implicit aspects.
  • TrustData: A custom dataset for trust prediction, comprising 2925 documents (60.5% long) with aspects like "ability," "dependability," "integrity," and "purpose."
  • Hyperpartisan: A dataset of news articles labeled as hyperpartisan or not, used for polarity prediction.

Key Insights and Ideas

  • The critical role of long-range context: The paper strongly emphasizes that for long documents, simply processing sentences in isolation or using short context windows is insufficient. DART's hierarchical attention mechanism is key to capturing these distant dependencies.
  • Implicit aspects are a major challenge: The SocialNews dataset highlights that many aspects are not explicitly stated, making sentiment analysis even harder. DART's ability to infer sentiment for these implicit aspects is a significant achievement.
  • Aggregation is more than just averaging: The two-level aspect aggregation mechanism is crucial. It allows the model to intelligently weigh the importance of different sentences and tokens, leading to a more accurate representation of aspect-specific sentiment.
  • Transformer limitations and DART's solution: Standard Transformers have a quadratic complexity with sequence length, limiting them to ~512 tokens. DART's architecture, while using Transformers, is designed to handle longer sequences by processing them hierarchically and aggregating information, overcoming this limitation.
  • Versatility beyond sentiment: The paper demonstrates that DART's core architecture can be effectively adapted for related tasks like trust and polarity prediction, showcasing its robust design.

Key Results

  • State-of-the-art performance: DART achieved state-of-the-art accuracy on document-level aspect-based sentiment classification across multiple datasets, including the challenging SocialNews dataset.
  • Superiority on long documents: DART's performance advantage over models like Longformer, Big Bird, and LongT5 becomes more pronounced as document length increases, particularly for documents exceeding 1024 or 2048 tokens.
  • Outperforming LLMs: DART, with 687 million parameters, outperformed GPT-4 (zero-shot and few-shot) on several benchmarks, suggesting that specialized architectures can still excel over general-purpose large language models for specific tasks.
  • Ablation study confirms component importance: Removing the interaction (Global Context Interaction Block) or aggregation (Aspect Aggregation Block) components led to significant drops in accuracy, underscoring their critical roles.
  • Trust and Polarity Prediction: DART achieved the best accuracy in trust and polarity predictions on the TrustData and Hyperpartisan datasets, with statistically significant improvements.

Limitations and Future Work

  • Single Entity Assumption: The current DART framework assumes each document pertains to only one entity. The authors acknowledge the need for a benchmark and model capable of handling aspect-based sentiment towards multiple entities within a single document.
  • Future Directions: The paper suggests extending DART's capabilities to handle aspect-based sentiment analysis involving multiple entities. The authors also plan to make their curated SocialNews dataset publicly available.

Summary updated

Abstract

The growing number of online articles and reviews necessitates innovative techniques for document-level aspect-based sentiment analysis. Capturing the context in which an aspect is mentioned is crucial. Existing models have focused on relatively short reviews and may fail to consider distant contextual information. This is especially so in longer documents where an aspect may be referred to in multiple ways across dispersed sentences. This work introduces a hierarchical Transformer-based architecture that encodes information at different level of granularities with attention aggregation mechanisms to learn the local and global aspect-specific document representations. For empirical validation, we curate two datasets of long documents: one on social issues, and another covering various topics involving trust-related issues. Experimental results show that the proposed architecture outperforms state-of-the-art methods for document-level aspect-based sentiment classification. We also demonstrate the potential applicability of our approach for long document trust prediction.
BibTeX
@inproceedings{yan2024modeling,
  author = {Zehong Yan and Wynne Hsu and Mong-Li Lee and David Bartram-Shaw},
  title = {{Modeling Complex Interactions in Long Documents for Aspect-Based Sentiment Analysis}},
  booktitle = {WASSA},
  year = {2024},
  doi = {10.18653/v1/2024.wassa-1.3},
}
Figure