Platforms that rely on volunteers, such as crowdsensing apps, demand-response programs and crowdsourcing markets, increasingly pay people different amounts. Low-cost participants get less and reluctant ones get more, so a fixed budget recruits more effort. The published gains are large, but almost all of them are measured inside simulators. In those simulators, people accept offers according to a sigmoid curve, a weighted-sum rule or a linear-quadratic utility. We propose to test those simulators where it matters. First, we collect a modest randomized dataset of real people responding to randomly assigned payments. Then we ask whether a pricing policy optimized inside each simulator actually earns what the simulator promises when it is scored on the real responses.
The central bet is about which question to ask of a simulator. "Does it fit the data?" is the wrong question for incentive design. The right question is "does it lead to the same pricing decisions?" Paying people differently is valuable because people differ in their minimum acceptable price, that is, their reservation price. A behavioral model that blurs a two-group population into one average group will misjudge that value, even if its overall fit looks acceptable. We will measure, for each family of simulator, how much real value is lost by trusting it. We will also measure how many real observations each simulator is worth, which is an exchange rate borrowed from synthetic-vision work.
Core Claim
Most adaptive incentive mechanisms report their results on synthetic behavior. First learn then earn fits a logistic choice model to a 50-person questionnaire and then reports 25% to over 150% quality gains on synthetic profiles. Identifying Influential Users in Unknown Social Networks uses real graphs but generates every user decision with an agent-based formula. Cost-Effective Incentive Allocation via Structured Counterfactual Inference is validated only on fully simulated data and an ImageNet semi-simulation. Fine-Grained User Profiling pairs a 100-user simulation with a 10-user trial. These papers motivate the proposal: none can say whether its policy would rank first, or even beat a flat rate, on real people.
Our claim has two parts. The first part is that a simulator's fitness for incentive design should be scored by decision regret. Decision regret is the real-world value lost by deploying the policy the simulator recommends instead of the best policy in the same class. We define it as
$$\Delta(S) = V_{\text{real}}(\pi^\star) - V_{\text{real}}(\pi_S),$$
where $\pi_S$ is the budget-feasible pricing policy that is optimal inside simulator $S$, $\pi^\star$ is the best policy in the same class, and $V_{\text{real}}$ is value estimated on held-out randomized human data.
The second part is a hypothesis about which simulators fail. We predict that single-population response models misstate the gain of paying people differently. This is the "flaw of averages" that Learning Real-Life Approval Elections documented for voting. We also predict that mixtures of response types close most of the gap. Finally, we predict that the monotonicity assumption used by structured counterfactual methods ("more pay never lowers participation or quality") breaks in a measurable fraction of repeated interactions.
Research Program
We first build the real response surface. This requires a new data collection, because the file contains no public dataset of humans responding to randomized personalized payment levels. ParticipAct varied notifications and gamification rather than money. The Double or Nothing MTurk studies varied payment structure rather than per-person amount.
We will run the study on a crowdsourcing platform. Workers receive task offers that differ in effort, with short and long tasks standing in for the walking distance in First learn then earn. Each offer carries a payment drawn uniformly from a fixed grid of levels. Each worker receives a sequence of offers over several sessions, so fatigue and history effects can appear. For every offer we record acceptance, completion and quality, where quality is scored on hidden gold questions as in Double or Nothing. Because payments are randomized with known probabilities, any pricing policy can be scored on held-out workers without bias using inverse-propensity weighting, plus a doubly robust variant to reduce variance.
Next, we fit a ladder of behavioral simulators on calibration pools of increasing size, drawn from workers kept separate from the evaluation pool. The ladder has five rungs, each of which supplies its method from a prior paper:
- A single logistic acceptance model in the style of First learn then earn.
- The weighted-sum adoption rule of the agent-based model in Identifying Influential Users, without its social term, since our tasks have no network.
- A monotone structured response model in the spirit of SIRE from Lopez et al..
- A finite mixture of logistic response types fit by EM, adapting the mixture approach of Learning Real-Life Approval Elections.
- The same mixture with a fatigue and history state.
Within each simulator we solve the same budget-constrained personalized pricing problem, using an identical optimizer and an identical policy class.
Finally, we score the simulators three ways.
Decision regret. We compute $\Delta(S)$ on the held-out pool.
Gain inflation. We compare the improvement over flat pricing that each simulator predicts with the improvement actually realized. This bears directly on the 23.18% and 25% to 150% type claims.
Exchange ratio. We find the number of real randomized observations a direct learner needs to match a simulator calibrated on $n$ observations. This is the incentive-design analogue of the roughly 30:1 synthetic-to-real frame ratio that On the Equivalency, Substitutability, and Flexibility of Synthetic Data found for tracking.
We also apply the authenticity check of Is this Model for Real?, which asks whether real data looks like a typical draw from the fitted model. We apply it to response curves to see whether this cheap test predicts decision regret.
Baselines
Flat-rate pricing (naive reference). Every worker receives the single payment level that maximizes held-out value under the budget. Any personalized policy must beat it on real value for discrimination to be worth its overhead. The literature's case against flat pricing is entirely simulated. One example is the 32.4% versus 66.5% activation rate reported by Wu et al., which rests on agent-based responses. Another is the 23.18% cost reduction reported by Tushar et al., which rests on a game-theoretic simulator. Flat pricing is therefore a real competitor here rather than a straw man.
Direct off-policy learning on real logs (strongest competitor). This skips simulators and learns prices directly from the calibration pool using CCPOvSIRE from Lopez et al.. On the ImageNet semi-simulation it reports an RMSE of 0.2408, against 0.2867 for BART and 0.3449 for GANITE. No reported result was found on Lacuna for CCPOvSIRE or any comparable method on real human incentive-response data. It will receive the same calibration data, the same policy class, the same budget and the same hyperparameter-search budget as the simulator pipeline, and it will be evaluated on the identical held-out pool. A simulator "beats" it at a given calibration size $n$ if its policy's held-out value is higher, with a bootstrap interval over workers that excludes zero.
Ablations. Two ablations isolate the proposal's key ingredients. The first compares the mixture simulator with and without its type structure, which tests heterogeneity. The second compares the structured models with and without the monotonicity constraint, which tests whether that assumption costs value on real data.
Why not just…?
Why not just learn prices directly from real data and forget simulators? This is the most serious threat, and the exchange-ratio experiment is designed to settle it. Simulators are used because designers must set prices before they have many logs, which is the cold-start problem flagged in Fine-Grained User Profiling and Wu et al.. Simulators only earn that role if a model calibrated on a small sample beats direct learning from the same sample. If direct learning dominates at every calibration size we test, we will report that as the main finding: simulators add nothing beyond a small randomized pilot.
Why not just judge simulators by held-out likelihood? Learning Real-Life Approval Elections found a 0.993 correlation between likelihood and distance from real data, so likelihood might suffice. Our reason to doubt it is specific. The value of personalized pricing depends on the tails of the reservation-price distribution, and likelihood weights those tails little. The experiment will rank the simulators by likelihood and by decision regret, then compare the two rankings. If the rankings agree, we will recommend likelihood as the cheaper metric and say so plainly.
Why not just validate on a richer synthetic generator? Making simulators more realistic without real anchoring is exactly the step that Chang et al. showed can backfire. In their work, naive synthetic motion hurt DanceTrack performance until crowding was modeled, after which HOTA rose from 68.8 to 73.5. Realism must be measured against a downstream outcome, and our randomized pool is that outcome.
Proposed Experiments and Decisive Checks
The primary experiment sweeps calibration sizes from a few dozen to several hundred workers. At each size we report decision regret, gain inflation and the exchange ratio for every simulator, against flat pricing and direct off-policy learning. All three metrics come with worker-level bootstrap intervals.
Check 1: heterogeneity. If the single-logistic and weighted-sum simulators show significantly larger regret than the mixture simulator, the heterogeneity hypothesis holds. If they do not, it fails.
Check 2: monotonicity. We estimate, from the randomized data, the fraction of workers or sessions in which higher pay lowered quality or later participation. We then test whether dropping the monotonicity constraint reduces regret. A null result would support the assumption behind Lopez et al. on real data.
Check 3: inflated gains. If simulator-predicted gains over flat pricing exceed realized gains by a margin larger than sampling error, published simulator-only gains should be read as upper bounds.
Robustness and cheap screening. A secondary replication will use a second task type, varying effort framing, to test whether simulator rankings are stable across tasks. We will also test whether the authenticity check from Is this Model for Real? predicts which simulators incur high regret, which would make it a cheap screening tool.
References
- First learn then earn: optimizing mobile crowdsensing campaigns through data-driven user profiling
- Identifying influential users in unknown social networks for adaptive incentive allocation under budget restriction
- Cost-Effective Incentive Allocation via Structured Counterfactual Inference
- Fine-Grained User Profiling for Personalized Task Matching in Mobile Crowdsensing
- Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality
- Feasibility of using discriminate pricing schemes for energy trading in smart grid