A Review Selection Approach for Accurate Feature Rating Estimation

Chong Long, Jie Zhang, Xiaoyan ZhuCOLING 2010 · OpenReview

Abstract: Online reviews often lack granular feature-level ratings, such as cleanliness or service quality, because users find multi-form entries tedious. While machine learning can estimate these ratings from text, the prevalence of short, noisy reviews limits accuracy. This paper proposes a selection approach based on Kolmogorov complexity and information distance to identify "comprehensive" reviews. By focusing on this high-quality subset, the authors improve automated feature rating estimation by over 12.5% and reduce aggregate score error by 23.2%.

Introduction

The core challenge in automated sentiment analysis for e-commerce and travel sites is the sparsity of feature-level data. While a user might provide an overall star rating, their text may only briefly mention specific attributes like "value" or "room quality." Standard models attempting to parse every review often struggle with this noise. The authors argue that not all reviewers are equally informative; identifying a subset of "expert" or comprehensive reviews can lead to more consistent and predictable ratings.

Methodology: Identifying Comprehensive Reviews

The proposed method leverages Information Distance and Kolmogorov complexity to rank reviews by their informativeness regarding specific features. Since Kolmogorov complexity is uncomputable, the authors use a practical approximation:

  1. Word Set Construction: For each feature (e.g., Value, Rooms, Service, Cleanliness), they identify "Core Words" (frequent terms), "Expanded Words" (synonyms via Google similarity), and "Dependent Words" (grammatically linked terms found via the Stanford Parser).
  2. Ranking and Selection: Reviews are ranked based on their information density relative to these word sets. The top 20% of reviews are designated as "comprehensive."
  3. Rating Estimation: Standard classifiers, including Support Vector Machines (SVM) and Bayesian Networks, are applied to this filtered subset to estimate 1-to-5 star ratings.

Experimental Results

The approach was tested on a TripAdvisor dataset covering hotels in Boston, Sydney, and Las Vegas.

Selection Performance

The selection algorithm proved highly effective at distinguishing comprehensive reviews from superficial ones. Across various cities and features, the system achieved F-scores generally exceeding 0.70, indicating a robust ability to identify high-quality content.

Performance metrics for the review selection algorithm across three cities and four features (Value, Rooms, Service, Cleanliness).

Estimation Accuracy and Consistency

A key finding is that "expert" reviewers—those who write comprehensively—tend to be more consistent. The standard deviation of ratings among these selected reviews was significantly lower than in the general population. More importantly, the accuracy of the machine learning models improved by more than 12.5% when trained and tested on the comprehensive subset.

Comparison of the average difference (error) between estimated and actual ratings for comprehensive reviews versus the full dataset.

When calculating aggregate feature ratings for a hotel, using the average of these "expert" estimates reduced the overall error by 23.2% compared to using the entire, noisier dataset.

Conclusion and Future Work

By prioritizing information-dense reviews over sheer volume, this approach provides a more reliable signal for both consumers and automated systems. Future research directions include developing sophisticated weighting schemes for different levels of comprehensiveness and automating the discovery of relevant features in new domains.

Summary updated

BibTeX
@inproceedings{long2010review,
  author = {Chong Long and Jie Zhang and Xiaoyan Zhu},
  title = {{A Review Selection Approach for Accurate Feature Rating Estimation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=azQ59c2jqch},
}
Figure