Evaluating Critiquing-based Recommender Agents

Li Chen, Pearl PuAAAI 2006

Introduction

As e-commerce catalogs grow in complexity, users often face choice paralysis. They rarely possess well-defined preferences at the start of a search; instead, they "construct" their preferences through interaction. Critiquing-based recommender agents address this by acting as digital sales assistants, allowing users to provide feedback on product attributes (e.g., "I want something cheaper"). This paper presents a comparative user study between two primary design philosophies: user-motivated critiquing and system-proposed critiquing.

The Critiquing Design Tension

The central question is whether a system should proactively suggest critiques or provide a flexible interface for users to define their own.

  1. Example Critiquing (EC): A user-motivated approach. It presents multiple examples and a "Value Comparison" interface where users manually "Improve," "Keep," or "Take any suggestion" for specific features.
  2. Dynamic Critiquing (DC): A system-proposed approach. It displays one item at a time and uses the Apriori algorithm to suggest "compound critiques"—bundles of changes like "Cheaper and Lower Resolution."
The Example Critiquing (EC) interface allows users to manually specify improvements for individual product attributes.

Methodology

The authors conducted a within-subjects study using two real-world catalogs: Tablet PCs (55 products, 10 features) and Digital Cameras (64 products, 8 features). The study evaluated both objective performance (accuracy, time, interaction cycles) and subjective user perceptions (cognitive effort, confidence, and purchase intention).

Results and Discussion

The study revealed a significant preference for the user-motivated EC approach. Despite EC requiring users to view more products and spend slightly more time, it achieved a "landslide victory" in several key areas:

  • Decision Accuracy: In a post-task check against the entire database, 86.1% of EC users stuck with their choice, compared to only 47.2% of DC users. This suggests the system-led approach often leads to sub-optimal settling.
  • The Control Paradox: Users perceived the EC system as less cognitively demanding. Having the "steering wheel" to navigate the search space reduced mental stress compared to evaluating automated suggestions.
  • Ineffective Compound Critiques: In the DC system, users ignored system-proposed compound critiques 72% of the time, preferring simple unit critiques. Automated tradeoff suggestions frequently failed to align with the user's mental model.
Subjective user ratings across four dimensions show that the user-motivated EC system outperformed the system-proposed DC system.

Conclusion

The findings underscore the importance of user control in recommender systems. While automated suggestions (DC) were occasionally praised for speed, they lacked the accuracy and trust-building capacity of the manual EC interface. Future work should explore hybrid systems that use data-driven insights to predict likely tradeoffs while maintaining a flexible, user-controlled interface.

Summary updated

BibTeX
@inproceedings{chen2006evaluating,
  author = {Li Chen and Pearl Pu},
  title = {{Evaluating Critiquing-based Recommender Agents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure