Abstract
Traditional human computation systems typically aggregate crowd responses to identify a single "objective truth." However, many real-world applications—from photo curation to personalized travel planning—rely on subjective preferences. This paper explores two protocols for injecting personalization into crowdsourcing: taste-matching and taste-grokking. By evaluating these methods on a dataset of aesthetic preferences, we demonstrate that crowd workers can effectively predict a requester's unique tastes by "grokking" their preferences through a limited set of examples. Our results show that taste-grokking not only outperforms taste-matching in accuracy but also transforms subjective tasks into engaging, objective challenges for workers.
Introduction
The "wisdom of the crowd" is a powerful tool for tasks with verifiable answers, such as image labeling or text transcription. In these scenarios, consensus serves as a proxy for correctness. However, this normative approach fails when the goal is personalized. For instance, determining which photos from a personal library are "best" or which vacation itinerary is most "appealing" is inherently non-normative.
While automated recommendation systems (e.g., collaborative filtering) address personalization, they are often hindered by the "cold-start" problem, requiring significant historical data to function. This paper investigates how paid crowd workers can provide on-demand personalization for new users or private datasets by leveraging human empathy and pattern recognition.
Methodology: Matching vs. Grokking
The researchers proposed and compared two distinct strategies for personalizing crowd tasks:
1. Taste-Matching
Taste-matching mirrors the logic of collaborative filtering. In this protocol, both the requester and the crowd workers rate a small "training set" of items. The system calculates the correlation between the requester’s ratings and those of the workers. Only those workers whose natural tastes align closely with the requester are hired for subsequent tasks. This method relies on the inherent identity and preferences of the worker.
2. Taste-Grokking
Taste-grokking treats crowd workers as "empathy engines." Instead of finding workers who already share the requester's taste, this protocol asks workers to learn and adopt the requester's perspective. Workers are shown a set of items previously rated by the requester and are then asked to predict how that specific requester would rate new, unseen items. Here, success depends on the worker's ability to infer underlying patterns from examples.
Experimental Setup
The protocols were tested using Amazon Mechanical Turk. The task involved rating 100 images of salt and pepper shakers on a scale of 1 to 4 stars. To establish a baseline and ground truth, 30 workers rated the entire set of 100 images. The researchers then simulated "requesters" by using subsets of one worker's ratings to train others via the matching and grokking protocols.
Results
The performance of each protocol was measured using Root-Mean-Squared Error (RMSE), where a lower score indicates higher predictive accuracy.
| Method | RMSE |
|---|---|
| Baseline (Unfiltered Crowd) | 1.59 |
| Taste-matching | 1.10 |
| Taste-grokking | 1.07 |
Both personalized protocols significantly outperformed the unfiltered baseline. While the performance gap between matching and grokking was relatively small, taste-grokking demonstrated lower variance and slightly higher accuracy, suggesting it is a more robust method for generating personalized data.
Discussion and Key Insights
The study revealed several critical advantages of the taste-grokking approach:
- Implicit Communication: Examples serve as a powerful communication tool. Requesters often find it difficult to articulate their aesthetic preferences (e.g., "I like minimalist designs with matte finishes"). By providing examples, they communicate these nuances implicitly, allowing workers to detect patterns that the requester might not have consciously identified.
- The "Puzzle" Factor: Qualitative feedback indicated that workers found taste-grokking more engaging than standard labeling. It transforms a repetitive task into a psychological puzzle, which may lead to higher worker retention and better data quality.
- Objective Subjectivity: Taste-grokking effectively converts a subjective task into an objective one for the worker. The "ground truth" is no longer the worker's own opinion, but the predicted opinion of the requester. This allows for the application of standard quality-control metrics, such as gold-standard questions and inter-rater reliability.
Conclusion and Future Work
This research demonstrates that the crowd can be successfully "tuned" to individual preferences. Taste-grokking, in particular, offers a promising path for personalized human computation without the need for massive historical datasets.
Future work will explore the application of these protocols to more complex domains, such as creative writing or complex planning. Additionally, the researchers aim to optimize "example selection"—identifying which specific training items most effectively communicate a user's taste to the crowd.
Summary updated