When using crowdsourcing to estimate continuous values—such as product prices, event probabilities, or sports point spreads—how many "gold standard" control questions should you give each worker?
Crowdsourcing is a powerful way to harness the "wisdom of crowds," but not all workers are created equal. Some are experts, some are consistently biased, and some provide highly noisy answers. To get accurate results, researchers and practitioners usually "seed" tasks with control items (questions where the true answer is already known) to score worker reliability.
However, this introduces a fundamental trade-off: every control question a worker answers is one less "real" target question they can complete. If you use too few control items, your estimate of the worker's skill is inaccurate. If you use too many, you waste your budget and gather fewer data points on the items you actually care about. This paper provides theoretical and empirical rules of thumb for finding the mathematical "sweet spot" for this allocation.
Estimation Methods and Worker Models
The researchers focused on estimating continuous quantities using Gaussian models to represent worker behavior. They specifically looked at:
- Bias-only models: Workers have a consistent offset (e.g., always overestimating prices).
- Variance-only models: Workers have different levels of precision or "noise."
- Bias-variance models: A combination of both systematic bias and random noise.
To aggregate the data, they analyzed two main types of consensus algorithms:
- Two-stage Estimator: The system first uses control items to calculate a worker's bias and reliability. It then uses those fixed scores to weight the worker's answers on the unknown target items.
- Joint Estimator: The system simultaneously estimates the worker's parameters and the true values of the target items using the entire dataset. This leverages the fact that workers' agreement with each other provides valuable information, even without ground truth.
Theoretical Insights: The Square Root Rule
The paper provides mathematical proofs for the optimal number of control items ($k$) given a worker's total task budget ($\ell$).
For the common Two-stage Estimator, the optimal number of control items scales with the square root of the total budget. Specifically, $k \approx \sqrt{a\ell}$, where $a$ is a constant based on the model type (e.g., $a=3$ for the bias-variance model).
In contrast, Joint Estimation is much more efficient. Because joint estimators "learn" about worker reliability from the target items themselves, they require significantly fewer control items. The optimal $k$ for joint estimation scales as $O(\ell / \sqrt{n_t})$, where $n_t$ is the total number of target items. As the number of target items grows, the optimal number of control items for a joint estimator drops dramatically—sometimes to just one, which is enough to anchor the system and fix the "unidentifiability" of bias.
Real-World Validation
The authors validated their theoretical proofs using both simulations and real-world datasets:
- Price Dataset: 155 undergraduate students estimated the prices of 80 household items.
- NFL Forecasting Data: 386 participants predicted the point spreads of 245 NFL games.
The empirical results closely matched the theoretical predictions. For instance, in the NFL dataset, the optimal allocation for the two-stage estimator followed the $\sqrt{\ell}$ prediction almost perfectly.
The datasets also highlighted the importance of choosing the right model. In the Price dataset, workers showed systematic bias (generally underestimating prices), making the bias-only model perform better than variance-only models. Conversely, in the NFL dataset, there was no systematic bias across the crowd, making the variance-only model superior.
Limitations and the Fragility of Sophistication
While joint estimators are mathematically superior and require fewer control items, they are "brittle." The paper highlights that if your model of worker behavior is wrong (model misspecification)—for example, assuming a worker has no bias when they actually do—the joint estimator's performance can degrade significantly. In these cases, the simpler two-stage estimator is much more robust.
Furthermore, the theory is built on Gaussian noise. While many real-world tasks can be transformed to fit this assumption, it may not apply directly to categorical tasks like image labeling without further extension. Ultimately, control items are not just useful for scoring workers; they are also crucial for determining which model to use in the first place.
Summary updated