TL;DR
This paper introduces MPQA 3.0, an expanded version of the MPQA opinion corpus, by adding entity and event target (eTarget) annotations. This new resource aims to enable more granular sentiment analysis, moving beyond document-level or aspect-level sentiment to pinpoint precisely what entities or events are the subject of sentiment, which is crucial for applications like question answering.
What Problem Does This Paper Address?
Sentiment analysis has evolved from document-level to sentence-level and aspect-level approaches. However, a significant gap remains in precisely identifying the specific entities or events that are the targets of sentiment, especially in diverse text genres beyond product reviews. Existing corpora often focus on product aspects or limited to sentiments expressed solely by the writer. This paper addresses the need for a richer annotation scheme that captures sentiments directed towards entities and events, enabling systems to answer questions like "Who or what is X negative/positive towards?" and to build more comprehensive knowledge bases about opinions.
Methods
The core contribution is the development of an annotation scheme for adding "eTarget" annotations to the existing MPQA 2.0 corpus. MPQA 2.0 already provides rich annotations for subjective and objective speech events, including nested source structures and sentiment attitudes. The new scheme extends this by identifying specific spans within the text that represent entities or events being targeted by these sentiments.
The annotation process involved:
- Extending existing annotations: eTargets are added to both sentiment attitudes and expressive subjective elements (ESEs).
- Defining eTarget types: eTargets can be entities or events, with a specific handling for negated events.
- Annotation strategy: Annotators identify eTargets that are either within the span of an existing MPQA 2.0 target or are new targets not previously captured. For ESEs, eTargets are more narrowly defined as the entities/events directly evaluated by the expression.
- Agreement study: An agreement study was conducted on four documents from the OPQA subset to evaluate the reliability of the new annotation scheme, yielding a good average F-measure.
Datasets
The primary dataset is the MPQA corpus, specifically an expanded version referred to as MPQA 3.0. This new version builds upon the existing MPQA 2.0 annotations by incorporating the newly defined eTarget annotations. The paper also mentions the OPQA subset of the MPQA corpus, from which documents were selected for the agreement study. The authors note that MPQA 3.0 will consist of 70 documents, including a subset of MPQA 2.0 and the OPQA subset.
Key Insights and Ideas
- Moving beyond "aspects": The paper highlights the limitations of aspect-level sentiment analysis, which often restricts targets to features of products or services. MPQA 3.0 aims to capture a broader range of targets, including entities and events, making it more applicable to general text.
- The importance of "who/what is affected": Precisely identifying the target of a sentiment is crucial for understanding the full meaning. For example, knowing someone is negative about "insulting" is different from knowing they are negative about the "Prophet" being insulted.
- Nested sentiment structures: The MPQA corpus's inherent support for nested sources and attitudes is leveraged to represent complex sentiment expressions, where sentiments about events can be embedded within sentiments about other events or entities.
- Distinction between attitude and ESE eTargets: The paper clarifies that eTargets for attitudes encompass all entities/events the attitude holds towards, while eTargets for ESEs are more narrowly focused on what the specific expression directly evaluates.
Key Results
- The paper presents an annotation scheme for adding eTargets to the MPQA corpus.
- A subset of the MPQA corpus has been annotated with eTargets.
- The agreement study showed good inter-annotator agreement, with an average F-measure of 0.82.
- The current corpus (MPQA 3.0) contains 1,029 ESEs, 1,287 attitudes, and 1,213 target spans from MPQA 2.0, augmented with 4,459 eTargets in total. This includes 1,366 eTargets added to ESEs, 1,608 to target spans, and 1,485 eTargets not associated with any target span.
Limitations and Future Work
- Focus on writer's sentiment: The current work, like its predecessor, primarily considers sentiments expressed by the writer, excluding sentiments attributed to other entities.
- No eTargets for objective speech events: As of the paper's writing, eTargets have not yet been added to objective speech events, although the structure is present.
- Future work: The authors believe MPQA 3.0 will be a valuable resource for developing entity/event-level sentiment analysis systems, facilitating NLP applications like Automatic Question Answering. The paper implicitly suggests further expansion of the corpus and refinement of the annotation scheme.
Summary updated