For a few dollars less: Identifying review pages sans human labels

Luciano Barbosa, Ravi Kumar, Bo Pang, Andrew TomkinsNAACL 2009 · OpenReview

Luciano Barbosa, Ravi Kumar, Bo Pang, Andrew Tomkins
Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (2009)

Abstract

Identifying review pages across the vast expanse of the web is a classic "needle-in-a-haystack" problem. While supervised learning models like SVMs are effective, they require expensive human-labeled training data that scales poorly. We present a "human-label-free" approach that combines a noisy text classifier with a decision tree exploiting website URL structures. Our hybrid method achieves performance nearly identical to state-of-the-art supervised models without requiring any manual labels.

Introduction

While dedicated review sites are easy to index, a significant portion of review content is "horizontal"—buried within general-purpose sites like the New York Times or Amazon. For search engines, identifying these pages is crucial for features like sentiment aggregation or "filter by reviews." However, manual labeling for supervised classification is slow, expensive, and fails to capture the web's diversity. This paper investigates whether a high-performing review detector can be built using only "noisy" labels generated automatically from URL heuristics.

Methodology: From Noise to Refinement

The authors propose a two-stage pipeline that moves from global noisy signals to refined site-specific signals.

1. Learning from Noise

To obtain training data for free, the system uses a simple heuristic: URLs containing the word "review" are treated as "noisy positives," while others are "noisy negatives." A Naive Bayes classifier is then trained on the text of these pages. The authors mathematically demonstrate that as long as the noisy positive set has a higher density of actual reviews than the negative set, the classifier correctly identifies the linguistic feature weights (e.g., words like "pros," "cons," and "rating") even if the magnitudes are distorted.

2. Exploiting Site Structure

Web developers typically organize sites logically. The authors use a Decision Tree to cluster URLs based on their tokens, such as path components and key-value pairs. If one page at site.com/reviews/item123 is a review, its neighbors in the same URL cluster likely are too.

3. The Hybrid Approach

The system combines the text-based noisy classifier with the site-structure clusters. If a cluster of URLs is "homogeneous" (i.e., most pages in that cluster are labeled as reviews by the text classifier), the labels are smoothed so that every page in that cluster is treated as a review.

Key Results

The model was evaluated against a gold-standard test set of 2,000 hand-labeled webpages from 40 diverse websites, ranging from small blogs to massive retailers.

Performance of the hybrid model configurations (u and w) across different thresholds, showing accuracy and F-measure.

The hybrid method ($hyb_{1/3}$) achieved a peak accuracy of 77.7%. As shown in the performance curves, the model is robust across a range of thresholds. This approach significantly outperformed "off-the-shelf" subjectivity detectors like OpinionFinder, which are computationally expensive and often struggle with the diverse language of the web.

Comparison of the proposed method (lwd) against baseline OpinionFinder (opf) metrics.

Most impressively, the label-free method came within 2% of the performance of a supervised SVM (79.5% accuracy) that had the advantage of being trained on human-labeled data. This demonstrates that linguistic "vibes" can be effectively learned from noisy data without sacrificing significant accuracy.

Conclusion and Future Work

By treating URLs as proxies for publisher intent and combining them with robust text classification, this method provides a web-scale solution for review identification. While the reliance on URL tokens is a potential limitation for sites with randomized paths, the approach offers a highly efficient alternative to manual labeling. Future work may explore incorporating visual layout features (CSS/HTML structure) to further refine detection.

Summary updated

BibTeX
@inproceedings{barbosa2009few,
  author = {Luciano Barbosa and Ravi Kumar and Bo Pang and Andrew Tomkins},
  title = {{For a few dollars less: Identifying review pages sans human labels}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  url = {https://openreview.net/forum?id=Hkbo17-dZH},
}

Loading related content...

Figure