When many hospitals, camera-trap operators or regional agencies pool data to train one shared model, the model they get is only as good as its ability to work at sites it has never seen. The WILDS benchmark shows how steep that drop is. On Camelyon17-WILDS, standard training on pooled data falls from roughly 93% accuracy at the hospitals it trained on to roughly 70% at new hospitals (WILDS). A platform running such a pool therefore has to decide whom to recruit and how to reward them. It has to make that decision without knowing where the model will eventually be deployed. Existing data-valuation rules answer the question badly in opposite directions. One family pays for data that looks like everyone else's, which penalizes the genuinely new domains that fix out-of-distribution failures. The other pays for data that looks unusual, which invites contributors to upload noise.
We propose to value each contributor by a cross-dataset validation game, a scheme in which every contributor is both a supplier and a judge. Each participating site keeps a private held-out slice of its own data. A site's contribution is scored by how much its training data improves the shared model on the other sites' private slices, and the scoring site's own training data is always removed from the model it judges. These per-judge scores are then combined with a robust lower-quantile rule. Data that helps the model transfer across sites earns money. Noise that helps nobody earns nothing. Copies of a common domain earn little, because the other judges already get that value from existing data. The platform needs no held-out test set of its own, because the contributors jointly supply one. That is the sense in which validation itself becomes co-created.
Core Claim
Leave-one-site-out cross-validation, used as a payment rule and not just as an evaluation protocol, fixes the conflict between two existing valuation ideas: rewarding data for conforming to the pool and rewarding it for being unusual. Our testable claim has two parts. First, on real multi-site benchmarks, these cross-site values track a site's true marginal contribution to accuracy at withheld deployment sites better than the alternatives do. The alternatives are distribution-similarity valuation, uniqueness-based valuation, Shapley values computed against pooled validation data, and payment by dataset size. Second, under this rule the obvious strategic manipulations lower the manipulator's payment. Those manipulations are injecting noise, duplicating data, copying a popular domain's style and submitting a biased validation slice.
Formally, let there be $K$ sites. Site $k$ submits training data $D_k$ and holds private validation data $V_k$. Define judge $j$'s game over coalitions $S$ of sites as
$$v_j(S) = \mathrm{acc}\big(f_{\cup_{k \in S\setminus\{j\}} D_k};\, V_j\big),$$
the accuracy on $V_j$ of a model trained on the coalition's data, with $j$'s own training data always excluded. Let $\mathrm{Sh}_i(v_j)$ be site $i$'s Shapley value in that game, meaning its average marginal gain over random joining orders. Site $i$'s payment weight is
$$\phi_i = Q_\alpha\big(\{\mathrm{Sh}_i(v_j)\}_{j \neq i}\big),$$
where $Q_\alpha$ is a lower quantile or trimmed mean taken across judges. Excluding the judge's own training data is what makes the measure cross-dataset. Taking a lower quantile protects against a few hostile judges. It also expresses a worst-case notion of generalization, in the same spirit as the uncertainty-ball objective of Distributionally Robust Data Valuation, but taken over real sites instead of an assumed $\chi^2$ radius.
Position Relative to Prior Work
Two valuation papers supply the contrast. Data Distribution Valuation values a vendor's whole data distribution by its maximum mean discrepancy, a kernel distance between distributions, to a uniform mixture of all vendors' preview samples. It justifies that reference as minimax-optimal under a model in which each vendor's data is the true distribution mixed with some contamination. The paper reports a Pearson correlation of 0.989 with downstream performance on CIFAR-10 versus CIFAR-100. By construction, however, a vendor whose domain differs from the mixture scores as "contaminated," even when that domain is exactly what the deployment sites need.
Distributionally Robust Data Valuation takes the opposite stance. It scores individual points by a training-free "model deviation" computed from neural tangent kernel gradients, which rewards rare, dissimilar points. The authors warn that this can overvalue outliers and garbage. Our proposal differs from both in using actual cross-site predictive transfer, measured on data the contributor cannot see, instead of a geometric proxy. It pays that cost in retraining. Both papers are passive, buyer-side tools, and neither studies what contributors do in response to the rule. We make the strategic response part of the evaluation.
The platform-economics papers set the economic framing and one real threat. Pricing for Collaboration Between Online Apps and Offline Venues shows that a platform must subsidize external partners before they will invest in shared infrastructure. It reports 58% revenue gains from a charge-with-subsidy tariff over linear pricing. Content Contributor Management and Network Effects in a UGC Environment estimates that a newly acquired seller is worth about 242 Euros against 94 for a buyer, with most of that value coming through network effects. Both motivate paying contributors, and paying heterogeneous contributors differently. In contrast, Optimal Pricing for Peer-to-Peer Sharing With Network Externalities proves that uniform pricing is asymptotically optimal when contributions are homogeneous scalar capacities. That result constrains us: if flat per-sample payment performs as well as cross-site valuation on real benchmarks, our proposal fails. We therefore test it directly.
Enhancing Compositional Generalization via Compositional Feature Alignment supplies part of the method. It shows that frozen CLIP and DINOv2 encoders struggle on unseen domain-class combinations. We reuse such frozen encoders with retrained linear heads, which makes the many retrainings needed for Shapley values affordable.
Research Program
The first component is the valuation engine. Each site's data is encoded once with a frozen DINOv2 or CLIP encoder, and every coalition model is a linear probe on those features, so a single retraining takes seconds. We estimate Shapley values by Monte Carlo sampling of joining orders, with a truncation rule that stops when marginal gains vanish. For iWildCam's 323 camera locations, we additionally group locations into roughly 30 to 50 contributor units by operator or region, and we report how sensitive the values are to the choice of grouping.
The second component is a ground-truth protocol. Every WILDS dataset has deployment sites that are officially out of distribution and that the mechanism never sees. A site's true value is defined as its Shapley value in a game whose only judge is that official out-of-distribution test split. We report the rank correlation between each method's computed value and this true value. This check is decisive because no method under test may touch the official test sites.
The third component is a strategic stress test. We simulate contributors who can each take one of several actions: submit honest data, submit it with duplicated samples, add label noise, pad with generic web-style images, or copy the dominant site's domain. A contributor acting as a judge can also submit a validation slice skewed toward its own domain. For every valuation rule we measure whether each manipulation raises or lowers the manipulator's payment. The best-responses are computed by exhaustive search over this small action space; we are not solving for an equilibrium of a continuous game.
The fourth component is budgeted recruitment. A platform with a fixed budget recruits sites greedily by value per unit of cost, and we measure the out-of-distribution performance of the final model.
Baselines
The naive reference points are random recruitment and payment proportional to dataset size. The size-based rule is also the flat uniform per-sample price that Optimal Pricing for Peer-to-Peer Sharing With Network Externalities proves asymptotically optimal for homogeneous contributions. It is the natural default for a commercial data pool, so it is the first thing any cleverer rule must beat.
The strongest existing competitor is the maximum-mean-discrepancy distribution valuation of Data Distribution Valuation. It needs no retraining, it has a minimax justification, and it reports ρ = 0.989 in its own setting. It gets the same frozen features, the same site partition, preview samples of the same size as our validation slices, and its kernel bandwidth tuned on the same budget as our quantile $\alpha$. The file contains no reported result for it at the level of sites on WILDS, and none on Lacuna for per-site valuation against WILDS out-of-distribution splits for any method. Beating it means a statistically higher Spearman correlation with the true out-of-distribution value, over bootstrap resamples of sites, on at least three of the four datasets, and a lower payment to manipulators in the stress test.
The second competitor is the model-deviation score of Distributionally Robust Data Valuation, summed over each site's points and computed on the same frozen features. The paper reports lower worst-case error than leave-one-out, TracIn and LAVA when training on the top 45% of points, but the file gives no numeric values. We expect it to do well on honest sites and to fail when sites inject noise.
The key ablation keeps everything in our method fixed and changes one element: judges score against pooled validation data that includes their own site. This is ordinary Shapley valuation with an in-distribution validation set. A second ablation replaces the lower quantile with a plain mean, to isolate robustness to hostile judges. If the first ablation matches the full method, the cross-dataset exclusion is not doing the work.
For reference points on model quality, the WILDS pooled-training numbers are about 93% versus 70% on Camelyon17 and Pearson r of 0.60 versus 0.45 on PovertyMap (WILDS). Budgeted recruitment succeeds if, at matched budget, it closes a measurable fraction of that gap relative to size-based recruitment.
Why not just…?
Why not just pay a flat price per sample? This is the most serious threat, because the uniform-pricing result is a theorem. That theorem assumes interchangeable contributions. The WILDS findings suggest that real sites are not interchangeable, since which hospitals appear in training determines out-of-distribution accuracy. Our recruitment experiment compares the two rules at equal budget. If flat pricing matches cross-site valuation there, we will report that the extra machinery is not worth it for recruitment and restrict our claim to robustness against manipulation, where flat pricing plainly rewards duplication.
Why not just use the training-free distribution-similarity score? It is cheap and well founded. However, its reference point is the pool's own mixture, so a hospital with an unusual staining protocol looks like contamination even when that protocol matches a deployment site. The ground-truth rank-correlation test on Camelyon17 and PovertyMap measures exactly this failure. The copy-the-dominant-domain manipulation tests whether the score rewards conformity.
Why not just run Shapley against a validation set the platform holds? The platform generally lacks one drawn from deployment sites. Any set it does hold is effectively in-distribution for some contributors, who then collect value from the overlap with their own data. Our first ablation is this baseline. The comparison with the full method shows directly whether excluding the judge's own data changes which sites are rewarded.
Why not just use point-level uniqueness scores? Its own authors warn that it overvalues outliers. In a paid setting, that warning becomes an incentive to upload noise. The label-noise and padding manipulations measure how much a noisy site gains under it compared with under our rule.
Decisive Checks
The experiments use four datasets: Camelyon17-WILDS with its hospitals as contributors, PovertyMap-WILDS with its countries, FMoW-WILDS with its regions, and iWildCam 2021 with grouped camera locations (iWildCam 2021).
The proposal succeeds only if three results hold. First, cross-site values must rank sites by their true out-of-distribution value better than distribution-similarity scores, uniqueness scores and pooled-validation Shapley. Second, each manipulation in the stress test must lower the manipulator's payment under our rule while raising it under at least one baseline. Third, budgeted recruitment by cross-site value must beat flat per-sample recruitment on out-of-distribution accuracy at matched cost.
Failure on the first check sinks the idea. Failure only on the third means the rule is a sound, manipulation-resistant way to settle payments but not a better recruiting tool, and the paper would say so. We also report the compute cost of the approach, including Monte Carlo variance against the number of retrainings, since retraining cost is the price paid for measuring real transfer instead of a geometric proxy.
References
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Distributionally Robust Data Valuation
- Data Distribution Valuation
- Pricing for Collaboration between Online Apps and Offline Venues
- Content Contributor Management and Network Effects in a UGC Environment
- Optimal Pricing for Peer-to-Peer Sharing With Network Externalities
- Enhancing Compositional Generalization via Compositional Feature Alignment
- The iWildCam 2021 Competition Dataset