Extracting and Visualizing Trust Relationships from Online Auction Feedback Comments

John O'Donovan, Barry Smyth, Vesile Evrim, Dennis McLeodIJCAI 2007

Introduction

In online auctions, trust is the primary currency. However, legacy reputation systems, such as eBay’s, are often "massively biased" due to reputation inflation. Users frequently leave positive star ratings even when dissatisfied to avoid retaliatory feedback, burying their true grievances within free-text comments. This paper introduces AuctionRules, an NLP-based algorithm designed to extract granular, feature-based trust metrics from these comments and visualize them as a social network to facilitate better-informed decision-making.

The AuctionRules Methodology

The authors developed a classification pipeline tailored for the specific linguistic constraints of online auctions. Unlike general sentiment analysis, the algorithm focuses on eight core "feature sets" salient to auctions: Item, Person, Cost, Shipping, Response, Packaging, Payment, and Transaction.

The AuctionRules NLP pipeline for feature-based sentiment extraction.

As illustrated in the pipeline, the system utilizes Porter stemming to normalize words and a specialized "stop-word" list to filter noise. It identifies intensifiers (e.g., "very"), articles, and negators (e.g., "not") to determine sentiment polarity. A critical component is the Query Expander, which uses an online thesaurus to find synonyms or antonyms when specific keywords are missing, significantly increasing the algorithm's coverage from 62% to over 90%.

Data and Evaluation

The researchers crawled eBay to collect 10,000 user comments, specifically from the "Egyptian antiques" domain. To establish a "gold standard," 1,000 comments were manually rated by humans on a 5-point Likert scale.

AuctionRules demonstrated superior performance, achieving 97.5% accuracy and outperforming seven popular classification algorithms (including Naive Bayes and C4.5) by up to 21%. Notably, the system achieved a 0% false negative rate, ensuring that good sellers are not unfairly penalized while reliably identifying problematic behavior.

Visualizing Multi-dimensional Trust

A key insight of this research is that trust is not binary. By breaking trust down into specific features, users can assess risks more accurately. The system calculates "interpersonal trust" between users and "contextual trust" based on historical performance.

Visualization of contextual and interpersonal trust metrics within a browser interface.

The visualization component integrates these metrics directly into the browsing experience. Users can view a breakdown of a seller's performance across different categories—such as packaging quality versus communication speed—alongside a social network graph where edge thickness represents trust value and length represents the volume of supporting evidence. This allows for a personalized trust model where a buyer might prioritize item quality over shipping speed.

Conclusion and Future Work

AuctionRules effectively uncovers the "silent" negatives hidden within inflated positive ratings. In tests, it identified that 16% of comments rated as 93-100% positive by eBay actually contained negative indicators. While the algorithm currently struggles with extreme noise and misspellings, it provides a significantly more useful scale for buyers. Future work will focus on refining trust propagation mathematics to better calculate trust across extended social networks in a commercial context.

Summary updated

BibTeX
@inproceedings{odonovan2007extracting,
  author = {John O'Donovan and Barry Smyth and Vesile Evrim and Dennis McLeod},
  title = {{Extracting and Visualizing Trust Relationships from Online Auction Feedback Comments}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
}
Figure