Street-level photos can stand in for physical measurements. A classifier looking at crowdsourced street-view frames can tell paved from unpaved roads across a continent with better than 0.9 accuracy. The maps built this way have holes where nobody happened to upload pictures, and those holes fall mostly in poorer and more remote places. There are two ways to fill a hole. One is to generate synthetic road images, which is cheap. The other is to pay people to go and photograph real roads, which is expensive. This proposal asks how a mapping operator with a fixed budget should split money between the two, region by region.
The core idea is that synthetic data's value is local and has to be measured. In some regions generated images teach the model almost as much as real ones. In others they teach nothing, or make the model confidently wrong. The operator should therefore spend a small amount buying a few real "audit" photos in each region to estimate how much synthetic data is worth there. It should then set the price it offers for further real photos according to the gap that synthetic data cannot close. The pricing itself is learned online with a bandit, an algorithm that tries different offers and shifts toward the ones that work. Two quantities drive the scheme: how willing contributors are to accept a given price, and how much a real image is worth compared with a synthetic one.
Core Claim
Pricing crowdsourced visual data by its marginal model value is an open problem in dynamic pricing. Existing pricing models treat a completed task as a uniform success or failure and ignore what the data point teaches. The "exchange rate" between synthetic and real data is also known to vary unpredictably from task to task. We claim these two gaps are the same problem.
The quantity that should set the price of a real image in region $c$ is the residual error left after synthetic data saturates there:
$$ g_c = e_c(\text{synthetic saturated}) - e_c(\text{real added}), $$
Here $e_c$ is the error of the visual proxy, the model that reads road conditions from images, measured on real held-out images from region $c$. It is computed either after adding synthetic images for $c$ until gains plateau, or after adding real images. If $g_c$ is near zero, real photos in $c$ are worth roughly the cost of the synthetic images they replace, and the operator should offer little. If $g_c$ is large, the operator should pay up.
The crucial and testable claim is about what can and cannot detect a large $g_c$. The model's own uncertainty cannot be trusted to do it, because training on unrealistic synthetic images tends to make a model confident in exactly the places it is wrong. Only real audit images reveal $g_c$. Consequently, a budget-constrained acquisition policy that prices audits and collection jointly should beat two-stage pipelines that generate synthetic data first and then run uncertainty-driven collection. The advantage should be concentrated in the worst-covered regions.
Research Program
The setting is road-surface mapping on the African crowdsourced street-view dataset, which contains 200,000 expert-verified Mapillary frames over 3,440 grid cells of 100 km by 100 km. The classes are Paved, Unpaved and a small Unknown class, with a Fleiss' kappa of 0.995. This dataset was built entirely from passive, unpaid uploads with no mechanism for filling spatial gaps. We use it as a replay environment: each cell's images form a pool that can only be "bought" by posting a priced task. Cells are deliberately starved at the start to mimic real coverage holes.
On the synthetic side, each cell gets a conditional image generator for road surfaces. We plan an off-the-shelf text-and-image-conditioned diffusion model (a generator that learns by gradually denoising random noise into samples), prompted with the class and a regional style reference. This generator is our own choice; the file cites no road-image generator. The design follows On the Equivalency, Substitutability, and Flexibility of Synthetic Data, which found a roughly 30:1 synthetic-to-real frame exchange rate for multi-person tracking. It also found that structural diversity mattered more than photorealism. That study motivates our premise that the rate is task- and region-dependent, but it measured one global rate offline. We instead estimate a per-cell rate online and spend money on the result.
The policy keeps three estimates per cell or per cluster of similar cells:
- Acceptance curve. The first estimate is the probability that a contributor accepts a task at each discrete price tier. It is learned with an upper-confidence-bound bandit, an algorithm that favors options whose payoff is either known to be good or still uncertain. This follows DBP-UCB from Incentivizing Users for Balancing Bike Sharing Systems, which used reward tiers as bandit arms and converged in under a day of live deployment. Their user costs were walking distances on an abstract graph. Ours are travel costs to places with no imagery.
- Synthetic learning curve. The second estimate is how much the cell's error falls as synthetic images are added, read off the audit images.
- Saturation gap. The third estimate is $g_c$, with a confidence interval that shrinks as audits accumulate.
At each step the operator either generates synthetic batches for a cell, posts an audit task, or posts a collection task at a chosen price. The choice maximizes expected error reduction per expected dollar. The expected dollar cost of a real image is the price divided by the estimated acceptance rate.
Two structural results from Finish Them!: Pricing Algorithms for Human Computation constrain the price set. Under a fixed budget, at most two prices are near-optimal. Prices should rise with remaining workload. Following Tuning Crowdsourced Human Computation, we model price as shortening the wait before someone takes a task, not the time it takes them to do it.
Worker behavior in the replay is necessarily simulated, so the proposal's own results face the real-world validation gap. We confront this directly. Acceptance curves are drawn from several families: linear-then-plateau, multinomial logit, and remoteness-dependent reservation wages. Each is calibrated loosely to the 60% acceptance reported for Mainz. Any claimed ranking of policies must hold across all of them. A small live pilot then checks the curves themselves.
As a secondary testbed for cross-geography shift, we use the three-domain sequence RTK, then KITTI, then CaRINA from Continual Cross-Dataset Adaptation in Road Surface Classification. The question there is whether $g_c$ is large precisely for the emerging-country CaRINA domain, where synthetic generators are least likely to know the local materials.
Baselines
The naive reference spends the whole budget at one fixed price on real images, distributed uniformly over uncovered cells, with no synthetic data. A second naive reference spends nothing on real images and fills every hole with synthetic data. Together these bound the problem from both sides.
The strongest existing method for the visual model on this dataset is a Swin Transformer trained on the full expert-labeled data. Zhou et al. report it reaching accuracy 0.924, with all five architectures above 0.90. This is a full-data ceiling rather than a budgeted competitor. We report how much of the gap from the starved starting point to 0.924 each policy closes at each budget. No reported result was found on Lacuna for budget-constrained or priced acquisition on this dataset, so all budgeted numbers will be new.
The strongest existing method for choosing synthetic data is the UCB curator of Multi-Armed Bandit Approach for Optimizing Training on Synthetic Data. It chooses among synthetic subsets during training, rewarded by validation-accuracy gains, and reports up to 10% accuracy improvement over FID-guided selection. It has no prices and no real collection, so we run it as the synthetic-selection component inside a two-stage competitor. That competitor first curates synthetic data for every cell with this bandit, then spends the remaining budget on real images at prices learned by DBP-UCB, targeting the cells where model predictive entropy is highest.
Every method gets the same generator and the same number of generated images per dollar. All use the same Swin backbone, the same total budget, the same simulated worker draws (common random numbers, so the methods face identical random worker behavior), and the same hyperparameter tuning budget on a held-out set of cells.
Beating the two-stage competitor means a higher area under the accuracy-versus-budget curve. It also means significantly better accuracy on the worst-decile cells at matched spend, across all worker-model families.
Two ablations isolate the key ingredient. The first removes audits and estimates $g_c$ from model uncertainty alone. The second keeps audits but uses a single global exchange rate instead of a per-cell one. If either ablation matches the full method, the claim that locally audited exchange rates should set prices is wrong.
Why not just…?
Why not just pay a flat price for images from every empty cell? It is simple and fair. Prior work suggests it wastes money in two ways. A fixed price either starves remote cells or overpays easy ones; Finish Them! reports a 36% cost reduction from dynamic over fixed pricing on Mechanical Turk ($3.20 against $5.00). Flat pricing also pays full price for real photos in cells where cheap synthetic images would do. The flat-price baseline measures how large both losses are here.
Why not just generate synthetic images everywhere and adapt the model? This is cheapest, and in uniform terrain it may be nearly enough. The reported 0.999 accuracy in entirely unpaved regions suggests some cells are easy. The tracking study, however, found that simple synthetic data actively hurt performance on its harder benchmark. Our synthetic-only arm, evaluated on real held-out images per cell, shows directly where substitution fails. If it matches the full method on worst-decile cells, the proposal has no reason to exist, and we will report that.
Why not just generate synthetic data first and then do uncertainty-driven collection? This is the real threat, and it is the two-stage baseline above. Our argument against it is the overconfidence mechanism: synthetic training may lower uncertainty in exactly the cells where it raises error. That is an empirical claim, and we test it directly. If the correlation between model entropy and true per-cell error after synthetic training is as high as before it, uncertainty is a usable proxy and audits are unnecessary. In that case we would drop the audit mechanism and recommend the two-stage pipeline.
Proposed Experiments
The main experiment sweeps total budget over a range spanning the starved start to the cost of buying every image. It reports accuracy-versus-budget curves, worst-decile cell accuracy, dollars per percentage point gained, and the fraction of spend going to audits. We repeat the experiment over the three worker-model families and over two generators of different quality, to check that the benefit grows as the generator worsens.
A diagnostic experiment measures the per-cell exchange rate and saturation gap exhaustively offline, using all available real images. It then asks how well the online audited estimates recover them after a few audits per cell, and whether the ranking of cells by audited gap matches the ranking by true gap.
The decisive checks are three. First, the entropy-versus-error correlation test, run before and after synthetic training. Second, the per-cell versus global exchange-rate ablation. Third, on the RTK, KITTI and CaRINA sequence, whether the audited gap is largest on CaRINA. Accuracy there is compared against the reported Less-Forgetful Learning result, which retained an AUROC of 0.9774 on the original data within 2.38% of joint training.
Finally, a small live pilot posts priced photo tasks to contributors in a few under-mapped areas. It tests whether the simulated acceptance curves rank prices correctly. If they do not, the replay conclusions about pricing, though not those about audit value, must be withdrawn.
References
- A Large Crowdsourced Street View Dataset for Mapping Road Surface Types in Africa
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- Incentivizing Users for Balancing Bike Sharing Systems
- Finish Them!: Pricing Algorithms for Human Computation
- Tuning Crowdsourced Human Computation
- Continual Cross-Dataset Adaptation in Road Surface Classification
- Multi-Armed Bandit Approach for Optimizing Training on Synthetic Data