TL;DR
This paper introduces the first bilingual (English/Spanish) annotated dataset of app store reviews using Appraisal Theory, a linguistic framework for analyzing subjectivity. This resource aims to bridge the gap in NLP's understanding of nuanced sentiment beyond simple polarity, enabling more sophisticated analysis of user opinions.
What Problem Does This Paper Address?
App stores are flooded with user reviews, which significantly influence product success. While Natural Language Processing (NLP) has made strides in sentiment analysis, most approaches focus on broad polarity (positive/negative). However, user opinions are far more complex, involving subtle evaluations, judgments, and emotional expressions. Existing NLP tools struggle to capture this richness, often treating all positive or negative feedback as equivalent. This paper addresses this limitation by creating a dataset annotated with Appraisal Theory, a linguistic framework that provides a detailed system for analyzing how speakers/writers express and negotiate opinions and attitudes. This allows for a more granular understanding of subjective language, moving beyond simple sentiment scores to capture the how and why behind user opinions.
Methods
The authors developed an annotation scheme based on Martin and White's (2005) Appraisal Theory, which categorizes subjective language into three main subsystems: Attitude, Engagement, and Graduation. They created a core tagset common to both English and Spanish and then an extended, more delicate tagset. To ensure the reliability of their annotations, they conducted rigorous agreement studies with independent annotators. These studies involved identifying spans of text, assigning coarse Appraisal tags, and finally, fine-grained annotation using the extended tagset. The annotation process was supported by the UAM Corpus Tool, a specialized software for corpus annotation.
Datasets
The paper introduces a newly compiled bilingual (English and Spanish) dataset of 250 app store reviews from Google's Play Store. This dataset was derived from a larger corpus of nearly 50,000 English and 37,000 Spanish reviews. The 250-review subset was carefully selected to ensure an equal distribution across languages, review lengths, polarity types (positive/negative), and app categories (applications, games, films, books, music).
Key Insights and Ideas
- Beyond Polarity: The research highlights that user reviews contain much more than just positive or negative sentiment. Appraisal Theory allows for the identification of specific emotional expressions, judgments about behavior and qualities, and how users engage with and position themselves relative to the text and other viewpoints.
- Linguistic Nuance in Reviews: App store reviews exhibit unique linguistic features, including abbreviations, emoticons, and sentence fragments, due to smartphone usage. These features, while challenging for traditional NLP, are rich in subjective meaning that Appraisal Theory can capture.
- Language-Specific Preferences: While both English and Spanish reviews utilize Appraisal features, there are notable differences. Spanish reviews show a higher preference for Attitude, particularly expressions of feelings and qualities, while English reviews lean more towards Engagement and Graduation, modulating certainty and using more nuanced expressions of contrast.
- Product Category Differences: The study found that applications and games tend to use more Engagement resources, whereas books and films show a higher use of Attitude categories.
- Positive vs. Negative Review Differences: Positive reviews tend to use more Appreciation and Happiness expressions, while negative reviews lean more heavily on Judgement (especially concerning propriety) and expressions of Dissatisfaction and Unhappiness.
Key Results
The annotation of the 250-review dataset revealed that Attitude was the most frequently annotated category (40.89%), followed by Engagement (35.64%) and Graduation (23.46%). Within these, Contraction (a subtype of Engagement) was the most frequent specific tag, likely due to the prevalence of negations and hypothetical situations. Appreciation (a subtype of Attitude) was also highly represented, reflecting the evaluative nature of reviews. Agreement studies showed high inter-annotator agreement (Kappa scores ranging from 0.49 to 0.96), indicating the robustness of the annotation scheme, though some categories like Graduation and specific subtypes of Attitude proved more challenging to distinguish consistently.
Limitations and Future Work
The authors acknowledge that some Appraisal features, particularly within Graduation and certain Attitude subtypes, proved difficult to annotate consistently, suggesting areas for refinement in the annotation scheme and guidelines. Future work includes extending the annotation to longer phrases and sentences, investigating the empirical validation of more delicate Appraisal features, and expanding the dataset to a wider range of products to identify broader patterns and groupings. The creation of this bilingual dataset is seen as a crucial step for developing more sophisticated machine learning models for large-scale, fine-grained subjective analysis in NLP.
Summary updated