Synthetic benchmarks are cheap, and real benchmarks are trustworthy. A general-capabilities test set generated by a language model can cost about $5, compared with $100,000+ for a hand-built one like GPQA. Nobody knows, though, whether a model's score on such a set says anything about how it performs on real inputs. We propose to stop treating synthetic scores as a replacement for real evaluation and use them as a correction signal instead. For each unlabeled real test item we generate a synthetic twin: a matched item whose answer is known by construction. The model's score on the twin serves as a cheap stand-in for its score on the real item. A small, actively chosen set of human-labeled real items then removes whatever bias the twins carry. The resulting estimate of real-world zero-shot accuracy is unbiased no matter how bad the twins are. It needs fewer human labels in proportion to how well the twins track reality.
The by-product matters as much as the savings. The fitted gap between twin scores and real scores, measured per model, directly measures how valid the synthetic benchmark is for that model. It shows whether synthetic evaluation quietly favors models that share ancestry with the generator. Zero-shot evaluation here means testing a model on a task with no task-specific training or examples. The open question at the intersection of these two directions is when a synthetic zero-shot score can be believed. This proposal answers it with a number and a confidence interval rather than an argument.
Core Claim
Consider a pool of $N$ unlabeled real evaluation items. Let $\ell_i$ be a model's loss on real item $i$, which is observed only if a human labels it. Let $s_i$ be the model's loss on that item's synthetic twin, which is always observed because the twin's answer is verified at generation time. We draw $m$ items for human labeling from a proposal distribution $q$ and estimate the real risk as
$$\hat{R} = \frac{\lambda}{N}\sum_{i=1}^{N} s_i + \frac{1}{m}\sum_{t=1}^{m}\frac{\ell_{i_t}-\lambda s_{i_t}}{N\,q(i_t)},$$
where $\lambda$ is a scalar weight on the twin signal. Whatever the twins' bias, the estimate is unbiased for the true mean real loss, because the importance-weighted second term exactly cancels the expected error of the first. Its variance scales with the variance of the residual $\ell_i-\lambda s_i$ rather than of $\ell_i$ itself. The variance-minimizing weight is $\lambda^\ast=\mathrm{Cov}(\ell,s)/\mathrm{Var}(s)$, so the label savings grow with the squared correlation between twin and real outcomes; this is the standard control-variate result. The claim is empirical, and it has two parts. First, item-matched verified twins give a proxy correlated enough with real outcomes to cut labeling cost several-fold on real zero-shot benchmarks. Second, the per-model residual reveals systematic distortions in synthetic benchmarks that plug-in synthetic scores hide.
Relation to Prior Work
Four papers motivate the proposal. Zero-shot Benchmarking generates whole benchmarks with a language model and scores responses with a language-model judge. It shows rank agreement with human judgments, but its authors state that the link to real deployment utility is uncharacterized. It also leaves open whether generator-family models are favored. We keep its generation machinery but never trust its scores alone. Familiarity shows that synthetic training data can inflate "zero-shot" named-entity F1 to 55–58.5 when labels overlap with the test set, compared with 28.7–38.0 under genuine label shift. Its overlap metric correlates only moderately with F1 ($r$ = 0.299–0.517), so overlap alone cannot certify an evaluation. No "Zero-Shot" Without Exponential Data shows zero-shot accuracy tracks the logarithm of pre-training concept frequency and drops by up to 20% on its tail benchmark Let It Wag!. A synthetic generator trained on the same web corpora will tend to sample exactly the familiar concepts. That is why a correction term measured on real items is needed. PROST shows that models exploit answer order and word association. This constrains twin design: twins must randomize option order and invert superlatives, or they will inherit the same shortcuts.
Two papers supply the method. Prediction-Powered Active Testing combines importance-weighted active testing with a control variate from a cheap proxy model, TabPFN-2.5, and samples where the proxy's residual is largest. It does this only for tabular regression, and its proxy is a model predicting labels on the same real points. Our proxy is different in kind. It is the evaluated model's own verified performance on a constructed counterpart item, which carries true ground truth rather than a guess. VeriSciQA supplies the twin-verification recipe. It extracts claims, generates questions, runs a caption-only blind test, and applies self-consistency filtering; only 3.0% of candidates pass, and the survivors are more useful than larger unverified sets. We reuse its filters to certify twin answers, with open models replacing the commercial ones where possible.
Synthetic Data as Validation supports the premise that synthetic validation can beat small real validation sets for model selection: out-of-domain DSC on FLARE'23 rises from 31.1% to 35.4%. It offers no way to know when the synthetic signal diverges from real risk, and our residual term provides one. SpaRRTa and the $M^3Act$ study in On the Equivalency, Substitutability, and Flexibility of Synthetic Data show procedural engines producing exact ground truth. The latter found behavioral complexity mattered more than photorealism, raising DanceTrack HOTA from 68.8 to 73.5. Both treat synthetic data as a separate substrate, never as a per-item counterpart to real test data. That pairing is the ingredient we add.
Research Program
The work has three parts. First, we build twin generators for three testbeds that each allow answers to be verified. In physical reasoning, PROST's template structure lets us generate twins that preserve the concept and question type of each real item but change the objects, option order, and superlative direction. Answers follow from the template, so they are exact. For entity recognition on CrossNER and MIT Movie, a language model rewrites each real sentence into a new sentence in the same domain. The entity spans are inserted programmatically, so the gold labels are known. For scientific-figure question answering on CharXiv and MathVista, we plot a figure programmatically with the same chart type and question category as the real item, from data we choose. The answer is then computed from that data, and the VeriSciQA-style blind test discards twins answerable without the image.
Second, we implement the estimator in three forms: uniform sampling, residual-driven active sampling in the style of PPAT, and a multi-proxy version. In the multi-proxy version, a language-model judge's guess on the real item and the twin score enter together as a vector control variate. Each form comes with confidence intervals whose coverage we check empirically.
Third, we use the fitted $\lambda^\ast$ and mean residual for each model as a synthetic validity profile. We test whether it differs systematically for models sharing the generator's family, and whether it grows on tail concepts. For the tail-concept test we use the Familiarity overlap score, extended from entity labels to twin-versus-real embedding similarity, as the stratifying variable.
Baselines
The naive reference is human labeling of $m$ uniformly random real items, with the sample mean as the estimate. It is unbiased and makes no assumptions, so any method must beat its error at equal $m$ to justify existing. The second reference is plug-in synthetic evaluation in the style of Zero-shot Benchmarking: report the mean twin score alone, with zero human labels. It is the cheapest competitor and the one practitioners actually use. We judge it by its bias against full-label truth, not its variance.
The third is standard active testing, the likelihood-based unbiased risk estimator without a proxy, as described in Prediction-Powered Active Testing. The strongest existing competitor is PPAT itself, with a language-model judge as the proxy predicting correctness on each real item. It is the strongest because it already combines unbiasedness, a control variate, and active sampling, and a strong judge may correlate with truth as well as twins do. The paper reports no per-dataset numerical variance reductions (no reported result found on Lacuna for PPAT's exact variance reduction), so we reimplement it. All methods share the same real pools, the same evaluated models, and the same label budgets from 25 to 400 items. The judge in PPAT gets the same generator model and inference budget used to make and verify twins, and any $\lambda$ tuning for either method uses the same held-out split.
Two ablations isolate the key ingredient. Unpaired twins assign each real item a random synthetic item of the same task; this tests whether item-level matching matters. Unverified twins skip the verification filters; this tests whether certified answers matter.
Beating the strongest baseline means two things. At every budget, our method must reach at least 1.5× lower mean squared error in real accuracy than judge-proxy PPAT, equivalently needing fewer labels for the same confidence-interval width. It must also achieve nominal 95% coverage. For scale, known real accuracies include UnifiedQA-3B at 46.7% on PROST (PROST) and the F1 ranges reported by Familiarity. Our full-label ground truth reproduces such numbers before any subsampling.
Why not just…?
Why not just use a language-model judge on the real items as the proxy, as PPAT would? This is the real threat. A frontier judge may predict correctness on real items better than any twin does. A judge scores the actual item but without verified truth. A twin carries verified truth but for a different item. So the judge's errors and the twin's errors should be partly independent, and it is plausible that they complement each other. The multi-proxy estimator tests this directly. If the twin adds no variance reduction on top of the judge, our contribution shrinks to the validity profile, and we will report that outcome as such.
Why not just trust the synthetic benchmark if it correlates with human rankings? Rank agreement on average hides per-model bias. Familiarity shows synthetic-to-benchmark overlap can double apparent zero-shot scores, and No "Zero-Shot" Without Exponential Data shows accuracy collapsing exactly where generators rarely sample. The plug-in baseline's bias against full-label truth, split by model family and tail stratum, will show whether that bias is large enough to flip rankings.
Why not just label a few hundred real items and skip synthetic data? When per-item outcomes are noisy, a few hundred labels leave confidence intervals wide enough that close models cannot be separated. Synthetic Data as Validation shows small real validation sets lead to worse model selection. The uniform-labeling baseline measures precisely how many extra labels twins save. If the twin-real correlation is low, the method gracefully reduces to this baseline, since $\lambda^\ast$ goes toward zero.
Why not just stratify sampling by a familiarity score? Stratification lowers variance only when strata differ in mean loss, and it uses no per-item signal. It remains a sensible cheap competitor, and we include a stratified-uniform variant to check whether residual-driven sampling adds anything beyond it.
Proposed Experiments
For each testbed we evaluate ten to fifteen open and commercial models spanning several families, one of which matches the twin generator. We compute full-label real accuracy as ground truth. We then simulate each method over 200 random seeds at each budget, reporting mean squared error, confidence-interval coverage and width, and Kendall rank correlation of the recovered model ordering against the full-label ordering.
The first decisive check is the label multiplier. It is the ratio of labels uniform sampling needs to match our interval width, compared against the same ratio for judge-proxy PPAT and for the multi-proxy combination.
The second decisive check is the validity profile. The generator-family model should show a significantly larger mean residual than other families, with twins overstating its real accuracy. The residual should also grow in the low-overlap stratum on the entity and physical testbeds. If neither pattern appears, synthetic evaluation is more trustworthy than feared, which is itself a useful finding.
The third check tests robustness to adversarial twins. We deliberately degrade twin quality, for example by removing option-order randomization so twins inherit the PROST shortcuts. Unbiasedness and coverage must survive while savings fall, confirming that the correction term, not twin quality, is what guarantees correctness.
References
- Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models
- Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
- PROST: Physical Reasoning about Objects through Space and Time
- Prediction--Powered Active Testing
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- Synthetic Data as Validation
- SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data