TL;DR
This paper presents the first results of annotating a Basque opinion corpus using Rhetorical Structure Theory (RST). By analyzing how discourse relations influence sentiment, they demonstrate the necessity of discourse information for accurate sentiment analysis, particularly for lexicon-based tools.
What Problem Does This Paper Address?
Sentiment analysis, the task of extracting subjective opinions from text, faces challenges at different levels: document-level classification, sentence-level subjectivity detection, and aspect/entity-level opinion targeting. While sentiment lexicons are useful, a crucial resource for deeper analysis is a corpus annotated with subjective information and, importantly, discourse structure. Understanding how different parts of a text relate to each other (discourse relations) can significantly impact the overall sentiment, either by strengthening, weakening, or even changing the polarity of an opinion. Existing sentiment analysis corpora often lack this discourse-level annotation, limiting their utility for tasks that rely on understanding text coherence and argumentation. This paper addresses this gap by annotating a Basque opinion corpus with RST, a framework for analyzing discourse structure, to investigate how these relations affect sentiment analysis.
Methods
The core methodology involves annotating a Basque opinion corpus using Rhetorical Structure Theory (RST). RST represents text as a tree structure where elementary discourse units (EDUs) are linked by coherence relations. The authors followed a three-step annotation process: text segmentation into spans (usually clauses), identification of clear relations between units, and linking relations until all EDUs are part of a discourse tree. They also employed the Basque version of the SO-CAL tool, a lexicon-based sentiment analysis tool, to extract sentiment valence from rhetorical relations. This involved using a Basque sentiment lexicon with 1,237 entries. Preliminary checks were conducted to assess the corpus's suitability for sentiment analysis, comparing first-person pronoun frequency and adjective presence against a neutral Basque corpus.
Datasets
The primary dataset used is the Basque Opinion Corpus, a newly created corpus of 240 opinion texts collected from various websites (newspapers, movie, and literature review sites). The corpus is multidomain (sports, politics, music, movies, literature, weather) and balanced in terms of domain representation and semantic orientation (positive/negative subjectivity). For the annotation and analysis presented in this paper, a subset of 70 texts was used, with 19 of these texts being doubly annotated by two annotators to measure inter-annotator agreement.
Key Insights and Ideas
- Discourse relations are crucial for sentiment analysis: The study highlights that simply analyzing individual sentences or even EDUs isn't enough. The way these units are connected through rhetorical relations significantly influences the overall sentiment, and lexicon-based tools like SO-CAL struggle without this information.
- Different rhetorical relations have distinct sentiment patterns: The paper shows that relations like CONCESSION and EVALUATION exhibit different distributions of sentiment-laden words. In CONCESSION, sentiment is relatively balanced between nucleus and satellite, while in EVALUATION, sentiment is more concentrated in the satellite.
- Basque SO-CAL needs discourse context: The analysis reveals instances where the Basque SO-CAL tool incorrectly assigns sentiment orientation because it doesn't account for the discourse function of rhetorical relations. This underscores the need for discourse-aware sentiment analysis.
- Annotation quality varies by relation type: Inter-annotator agreement was lower for certain rhetorical relations (e.g., ELABORATION, EVALUATION) compared to others, suggesting these are more open to interpretation or require more nuanced annotation guidelines.
Key Results
- Inter-annotator agreement: Manual evaluation of rhetorical relation types yielded an agreement of 39.81%. Automatic evaluation provided higher agreement (above 0.5) for aspects like constituent identification and nuclearity, but lower for relation type.
- Sentiment distribution: In CONCESSION relations, sentiment words appeared in nuclei (47.21%) and satellites (52.79%) with similar frequency, though satellites had a slightly higher proportion. For EVALUATION relations, sentiment words were more concentrated in satellites (55.00%) than nuclei (45.00%), with a notable exception in the weather domain.
- SO-CAL limitations: The study identified specific examples where SO-CAL failed to correctly predict the sentiment of a rhetorical relation due to a lack of discourse processing.
Limitations and Future Work
Limitations: * The study only annotates a portion of the Basque Opinion Corpus (70 out of 240 texts). * Inter-annotator agreement for rhetorical relation types was relatively low (39.81%), indicating room for improvement in annotation consistency. * The automatic evaluation of agreement was more strict and sometimes failed to compare relations due to minor differences in subconstituent descriptions.
Future Work: * Develop more comprehensive annotation guidelines to improve reliability and consistency. * Annotate the entire Basque Opinion Corpus. * Continue analyzing how subjective information is distributed across different rhetorical relations. * Explore assigning weights to EDUs within rhetorical relations to better model their sentiment impact, especially for lexicon-based tools.
Summary updated