TL;DR
This paper introduces DART, a novel hierarchical Transformer architecture designed to tackle the challenge of aspect-based sentiment analysis in long documents. By effectively capturing long-range dependencies and aggregating information at multiple granularities, DART significantly outperforms existing methods, even large language models like GPT-4, on complex sentiment classification tasks.
What Problem Does This Paper Address?
The internet is awash with user-generated content, from product reviews to news articles. Understanding the sentiment expressed towards specific aspects within this content is crucial for businesses and researchers alike. However, existing aspect-based sentiment analysis (ABSA) models often struggle with long documents. In lengthy texts, an aspect might be mentioned indirectly, across many sentences, or even in conflicting ways. Traditional models, often trained on shorter texts, fail to capture these distant contextual cues, leading to inaccurate sentiment predictions. This paper aims to bridge this gap by developing a model that can effectively process and understand sentiment in long documents, considering the nuanced interplay of information across numerous sentences.
Methods
The core of this work is the DART (Document-level Aspect-based Representation Transformer) framework. DART employs a hierarchical approach with several key components:
- Sentence Encoding Block: This initial stage uses a pre-trained language model (like BERT or BigBird) to generate embeddings for each sentence, crucially incorporating the target aspect into the input. This allows the model to start understanding sentences in relation to specific aspects.
- Global Context Interaction Block: This is where DART tackles long-range dependencies. It uses two Transformer encoders:
- The first encoder uses self-attention on the
[CLS]tokens of each sentence to capture inter-sentence relationships and create contextually enriched sentence representations. - The second encoder refines these representations by combining the context-enriched
[CLS]tokens with the original sentence token embeddings, allowing each sentence's representation to be influenced by both its local context and the broader document context. - Aspect Aggregation Block: This block is vital for synthesizing the information. It uses a two-level attention mechanism:
- Local Aggregation: It weighs the importance of each token within a sentence based on its relevance to the aspect and the overall sentence context.
- Global Aggregation: It then performs a global attentive pooling over the sentence-level aggregated representations to create a unified document representation for the specific aspect. This ensures that sentences more relevant to the aspect have a greater influence.
- Sentiment Classification Block: Finally, a two-layer MLP takes the aggregated document representation and predicts the sentiment polarity (positive, negative, etc.) for the given aspect.
Datasets
The authors evaluated DART on several datasets:
- BeerAdvocate: A dataset of beer reviews with ratings for aspects like "feel," "look," "smell," and "taste."
- TripAdvisor: A dataset of hotel reviews with ratings for aspects such as "value," "room," "location," and "cleanliness."
- SocialNews: A newly curated dataset from news articles related to social issues, focusing on implicit aspects like "crime-justice," "digital-online," "economic issues," "health," "human rights," and "work." This dataset is notable for its inclusion of long documents and implicit aspects.
- TrustData: A custom dataset for trust prediction, comprising 2925 documents (60.5% long) with aspects like "ability," "dependability," "integrity," and "purpose."
- Hyperpartisan: A dataset of news articles labeled as hyperpartisan or not, used for polarity prediction.
Key Insights and Ideas
- The critical role of long-range context: The paper strongly emphasizes that for long documents, simply processing sentences in isolation or using short context windows is insufficient. DART's hierarchical attention mechanism is key to capturing these distant dependencies.
- Implicit aspects are a major challenge: The SocialNews dataset highlights that many aspects are not explicitly stated, making sentiment analysis even harder. DART's ability to infer sentiment for these implicit aspects is a significant achievement.
- Aggregation is more than just averaging: The two-level aspect aggregation mechanism is crucial. It allows the model to intelligently weigh the importance of different sentences and tokens, leading to a more accurate representation of aspect-specific sentiment.
- Transformer limitations and DART's solution: Standard Transformers have a quadratic complexity with sequence length, limiting them to ~512 tokens. DART's architecture, while using Transformers, is designed to handle longer sequences by processing them hierarchically and aggregating information, overcoming this limitation.
- Versatility beyond sentiment: The paper demonstrates that DART's core architecture can be effectively adapted for related tasks like trust and polarity prediction, showcasing its robust design.
Key Results
- State-of-the-art performance: DART achieved state-of-the-art accuracy on document-level aspect-based sentiment classification across multiple datasets, including the challenging SocialNews dataset.
- Superiority on long documents: DART's performance advantage over models like Longformer, Big Bird, and LongT5 becomes more pronounced as document length increases, particularly for documents exceeding 1024 or 2048 tokens.
- Outperforming LLMs: DART, with 687 million parameters, outperformed GPT-4 (zero-shot and few-shot) on several benchmarks, suggesting that specialized architectures can still excel over general-purpose large language models for specific tasks.
- Ablation study confirms component importance: Removing the interaction (Global Context Interaction Block) or aggregation (Aspect Aggregation Block) components led to significant drops in accuracy, underscoring their critical roles.
- Trust and Polarity Prediction: DART achieved the best accuracy in trust and polarity predictions on the TrustData and Hyperpartisan datasets, with statistically significant improvements.
Limitations and Future Work
- Single Entity Assumption: The current DART framework assumes each document pertains to only one entity. The authors acknowledge the need for a benchmark and model capable of handling aspect-based sentiment towards multiple entities within a single document.
- Future Directions: The paper suggests extending DART's capabilities to handle aspect-based sentiment analysis involving multiple entities. The authors also plan to make their curated SocialNews dataset publicly available.
Summary updated