Abstract: Online reviews often lack granular feature-level ratings, such as cleanliness or service quality, because users find multi-form entries tedious. While machine learning can estimate these ratings from text, the prevalence of short, noisy reviews limits accuracy. This paper proposes a selection approach based on Kolmogorov complexity and information distance to identify "comprehensive" reviews. By focusing on this high-quality subset, the authors improve automated feature rating estimation by over 12.5% and reduce aggregate score error by 23.2%.
Introduction
The core challenge in automated sentiment analysis for e-commerce and travel sites is the sparsity of feature-level data. While a user might provide an overall star rating, their text may only briefly mention specific attributes like "value" or "room quality." Standard models attempting to parse every review often struggle with this noise. The authors argue that not all reviewers are equally informative; identifying a subset of "expert" or comprehensive reviews can lead to more consistent and predictable ratings.
Methodology: Identifying Comprehensive Reviews
The proposed method leverages Information Distance and Kolmogorov complexity to rank reviews by their informativeness regarding specific features. Since Kolmogorov complexity is uncomputable, the authors use a practical approximation:
- Word Set Construction: For each feature (e.g., Value, Rooms, Service, Cleanliness), they identify "Core Words" (frequent terms), "Expanded Words" (synonyms via Google similarity), and "Dependent Words" (grammatically linked terms found via the Stanford Parser).
- Ranking and Selection: Reviews are ranked based on their information density relative to these word sets. The top 20% of reviews are designated as "comprehensive."
- Rating Estimation: Standard classifiers, including Support Vector Machines (SVM) and Bayesian Networks, are applied to this filtered subset to estimate 1-to-5 star ratings.
Experimental Results
The approach was tested on a TripAdvisor dataset covering hotels in Boston, Sydney, and Las Vegas.
Selection Performance
The selection algorithm proved highly effective at distinguishing comprehensive reviews from superficial ones. Across various cities and features, the system achieved F-scores generally exceeding 0.70, indicating a robust ability to identify high-quality content.
Estimation Accuracy and Consistency
A key finding is that "expert" reviewers—those who write comprehensively—tend to be more consistent. The standard deviation of ratings among these selected reviews was significantly lower than in the general population. More importantly, the accuracy of the machine learning models improved by more than 12.5% when trained and tested on the comprehensive subset.
When calculating aggregate feature ratings for a hotel, using the average of these "expert" estimates reduced the overall error by 23.2% compared to using the entire, noisier dataset.
Conclusion and Future Work
By prioritizing information-dense reviews over sheer volume, this approach provides a more reliable signal for both consumers and automated systems. Future research directions include developing sophisticated weighting schemes for different levels of comprehensiveness and automating the discovery of relevant features in new domains.
Summary updated