Free References, Bounded Information: Multi-Reference Surrogates for Private Text Benchmarks

When a text-generation benchmark is built from private data, such as clinical notes, student answers, or enterprise email, outside groups cannot check published claims. A common workaround is to release a synthetic stand-in generated under differential privacy (DP), a guarantee that bounds how much any single private record can influence the output. One property of DP comes almost for free: once a generator has been trained privately, drawing more samples from it costs no extra privacy budget. A data owner could therefore release a stand-in test set with five, twenty, or a hundred synthetic reference answers per input. This matters because the multi-reference literature shows that adding references sharply improves how well automatic metrics track human judgment, up to roughly 5–8 references.

This proposal argues that the free references are not free in information. Every reference comes from the same noise-perturbed model, so they are correlated. Their usefulness should plateau earlier and lower than independent human references do, and the plateau should drop as the privacy budget tightens. We propose to measure that plateau in terms of reproducibility rather than human correlation. The question is whether "system A beats system B" on the released stand-in agrees with the same comparison on the private set with real human references. We also propose a cheap estimator of the "effective number of distinct references", so a data owner can predict where the stand-in stops improving before releasing it.

Core Claim

The saturation results in the literature measure correlation with human scores, averaged over systems. They come from human or strong-LLM references, which are fairly independent of each other. These studies include - Quality and Quantity of Machine Translation References for Automatic Metrics, with gains up to about 7 references on English–Czech; - Evaluation of Question Generation Needs More References, saturating near 5 ChatGPT paraphrases; - Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References, plateauing near 8; - A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing, where most gain comes within 5 of 20.

None of them asks how saturation behaves when references must pass through a privacy mechanism. None asks whether saturation in correlation implies saturation in the stability of pairwise conclusions, which is what reproducibility needs.

Our claim has three parts:

  1. The pairwise disagreement rate between stand-in and private evaluation falls with the number of references per input, $N$, and then levels off at a floor.
  2. That floor rises as $\epsilon$ (the DP budget) shrinks.
  3. Where the floor sets in is predicted by a coverage estimate of how many distinct references the private generator can actually produce.

We define the disagreement rate as

$$F(N,\epsilon)=\frac{1}{|P|}\sum_{(a,b)\in P}\mathbf{1}\big[\operatorname{sign}(s^{\text{sur}}_{N,\epsilon}(a)-s^{\text{sur}}_{N,\epsilon}(b))\neq \operatorname{sign}(s^{\text{priv}}(a)-s^{\text{priv}}(b))\big],$$

The symbols are:

  • $P$ is the set of system pairs whose private-set difference is statistically significant;
  • $s^{\text{sur}}_{N,\epsilon}$ is a system's metric score on the stand-in with $N$ references generated at budget $\epsilon$;
  • $s^{\text{priv}}$ is its score on the private set using all available human references.

Research Program

We simulate privacy on public multi-reference benchmarks. Each corpus is treated as if it were private. This gives us ground truth that a real private set would hide.

For each corpus, we fine-tune an open language model under DP on the "private" training split, at $\epsilon\in\{1,4,8,\infty\}$. The model learns to produce an input together with multiple outputs. We then sample stand-in inputs and, for each, $N\in\{1,2,5,10,20,50\}$ references.

We assemble a pool of runnable systems spanning a range of quality, including closely matched pairs, because close pairs are where flips happen. Every system is run on both the private test inputs and the stand-in inputs. Each is scored with BLEU, chrF, BERTScore, and COMET/BLEURT where applicable, under multi-reference aggregation by maximum and by average.

The private gold standard is the ranking under all human references. Where human system-level judgments exist, we check that this gold ranking itself agrees with humans.

The coverage estimator reuses the method of Inherent Biases in Reference-based Evaluation for Grammatical Error Correction. That paper applied an unseen-species estimator (UNSEENEST) to count how many valid corrections exist for a sentence, finding over 1,000 for short sentences. We apply the same idea to the DP generator's samples. Samples are clustered at a paraphrase-equivalence threshold. We then estimate how many distinct clusters the generator would produce in the limit, giving $N_{\text{eff}}(\epsilon)$. The prediction is that $F$ stops falling once $N$ exceeds a small multiple of $N_{\text{eff}}$.

Two prior papers act as constraints on the design.

SynBench found two problems with DP text generation:

  • apparent utility is inflated by overlap between benchmark text and the generator's pretraining corpus;
  • fidelity (MAUVE) collapses at $\epsilon\le 4$.

DP-Rewrite showed the same illusion in miniature. Snips-pretrained, Snips-rewritten reached F1 0.94 at $\epsilon=1000$, but this fell to 0.20 when the rewriting was done on ATIS.

We therefore audit n-gram overlap between each benchmark and the base model's pretraining data. Every result is reported with a control where the generator is trained on one corpus and evaluated against another. That way, a low $F$ cannot be explained by the generator having memorized the benchmark.

The benchmarks are:

  • the DailyDialog augmented test set (1,000 contexts, 5 human references);
  • the Quiz Design question-generation set;
  • English–Czech WMT2020 with its re-annotated reference variants;
  • WMT19 metrics data, where human system scores anchor the gold ranking;
  • the crowdsourced GEC/simplification corpus with 2,500 references, used to validate the coverage estimator against human diversity.

Baselines

Naive single-reference stand-in. The do-nothing point is the release format a benchmark like SynBench implies: one DP-generated target per synthetic input. Systems are scored on it with the same metrics, run on identical stand-in inputs with the same generator checkpoint. Our proposal beats it only if multi-reference stand-ins reduce $F$ by a margin larger than the bootstrap interval over inputs. For context, gains in human correlation from adding human references are reported:

Non-private paraphrase expansion. The strongest existing method for adding references is to paraphrase the single private reference with a strong LLM, following Evaluation of Question Generation Needs More References, which reports 92% meaning preservation. On WMT19, A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing reports a +0.024 average Pearson gain from 5 sampled paraphrases. This method leaks private content, so it is not a legitimate release. It serves as the ceiling for what any synthetic multi-reference set can do at a given $N$. We run it with the same LLM backbone and the same $N$ grid, and our DP stand-ins are judged by how much of the gap between the naive baseline and this ceiling they close. No reported result was found on Lacuna for pairwise disagreement rates of any synthetic or DP stand-in against a private gold ranking, so these numbers will be new.

Owner-run evaluation. The data owner scores submitted systems on the private set, as in the enclave or data-use-agreement model. This gives $F=0$ by construction and is the reference point we are trying to reach without the owner's involvement in every comparison.

Fixed-$N$ rule. We compare our estimator-chosen $N$ against simply releasing 5 or 8 references, the rule of thumb from prior saturation studies. The comparison is made at equal generation cost.

Correlation ablation. This ablation isolates the key ingredient. Instead of sampling all $N$ references from one DP model, we split the private data into disjoint shards. A separate DP model is trained per shard at the same $\epsilon$, which is valid by parallel composition because each record touches only one model. One reference is drawn per shard. If the plateau comes from shared-model correlation, sharded references should push the floor down despite each model seeing less data.

Why not just…?

Why not just use a reference-free LLM judge on the synthetic inputs and skip references altogether? This is the most serious threat. Released references would matter less if a judge scored outputs well without them. The file offers indirect evidence only. Learned metrics such as BLEURT and BERTScore still gained from multiple references in question generation and in MT (Quality and Quantity of Machine Translation References for Automatic Metrics), which suggests references carry information that semantic models do not recover alone. That evidence does not cover LLM judges. We therefore include a reference-free judge, and a judge given the $N$ DP references, as arms with the same $F$ measurement. If the reference-free judge matches the best multi-reference stand-in, the conclusion is that DP releases should ship inputs only, and we will report that.

Why not just generate hundreds of paraphrases with a bigger model? Without privacy this leaks the private references. With privacy it becomes our own setting, and the question is whether volume helps. Two findings suggest it will not:

CorporateBench further notes that generator-specific style biases its synthetic corpora. Our $N=50$ arm tests the volume argument directly: if $F$ keeps falling well past $N_{\text{eff}}$, the volume argument wins.

Why not just release de-identified data or rely on the owner running evaluations? MIMIC-III shows that de-identified release is possible. It also shows that it takes heavy infrastructure, including scrubbing 18 HIPAA identifiers and data-use agreements. Commercial settings such as Domain Adaptation for Enterprise Email Search do not release data at all. Owner-run evaluation is sound but makes every comparison depend on the owner. Our goal is the narrower complement: a released artifact that lets outsiders iterate and pre-check claims. The owner-run arm marks how close the artifact gets.

Proposed Experiments

Main experiment. The central result is the curve family $F(N,\epsilon)$ on each benchmark, with bootstrap intervals over inputs and system pairs. It is reported alongside the paraphrase ceiling, the naive single-reference baseline, and the judge arms.

First check: does a privacy floor exist? At $\epsilon=\infty$, $F$ should approach the paraphrase ceiling as $N$ grows. At $\epsilon\le 4$, it should level off at a strictly higher floor. If the floors coincide across budgets, the claim that privacy limits the information in references is false.

Second check: does the estimator predict the plateau? The knee of each curve, defined as the $N$ beyond which further references reduce $F$ by less than the bootstrap interval, should track $N_{\text{eff}}(\epsilon)$ across budgets and corpora with rank correlation clearly above zero. If it does not, the estimator adds nothing over the fixed-5 rule.

Third check: is the floor caused by correlation? In the sharded ablation, the floor should be lower than with single-model sampling at the same $\epsilon$. If it is not, the floor comes from information lost to noise rather than from correlation among references. That outcome would change the recommendation from "diversify generators" to "spend budget on fewer, better references."

Fourth check: is the result real or contamination? Every positive result must hold in the setting where the generator is trained on one corpus and evaluated against another, and on corpora with low measured pretraining overlap.