Rank-Faithful Surrogates: Distilling Private Datasets into Public Stand-ins That Preserve Method Rankings

When a hospital or company reports that its new model beats prior work on data nobody else can see, the claim cannot be checked. Outside researchers also cannot build on the result, because they cannot test their own ideas on that data. Knowledge distillation is the technique of training a small "student" model to imitate a large "teacher." It offers a partial escape. The data holder trains a teacher on the private records and uses it to label public or synthetic inputs, then releases that labeled set instead of the records. Prior work judges such stand-ins by one question: how accurate is a student trained on them? For reproducibility that is the wrong question. A researcher using a stand-in needs to know whether "my method beats yours" on the stand-in still holds on the private data.

This note proposes to make rank preservation the explicit target. The holder privately runs a small set of reference training recipes, so it knows how they rank on the real data. It then builds the released stand-in, which we call a surrogate, so that the surrogate reproduces that ranking. Fidelity is then tested on recipes the holder never used during construction. The study also tests a specific risk. The teacher's soft labels are the full probability vectors it outputs, not just its top answer. Such labels are known to inflate how good distilled data looks. We expect them to also flatten the differences between methods, which is exactly what a ranking tool must not do.

Core Claim

Existing private-distillation methods report high student accuracy. Examples are 82.1% on CIFAR-10 at privacy budget ε = 1.0 for reverse k-nearest-neighbor labeling (Fine-grained Private Knowledge Distillation), 81.69% under differential privacy for RONA (Private Model Compression via Knowledge Distillation), and 78.9% data-free for CAKE (Deep Classifier Mimicry without Data Access). None of these papers measures whether the data they produce preserves the ordering of competing methods.

We claim three things. First, the label format and the choice of inputs that maximize student accuracy are not the ones that maximize rank fidelity. Second, a surrogate calibrated against a few privately scored reference recipes generalizes its ranking to unseen recipes better than an uncalibrated teacher-labeled set. Third, both effects can be measured on simulated private settings where ground truth is available. If teacher-labeled random public data ranks methods as well as the calibrated surrogate, the measurement study and the soft-label finding remain the contribution. The calibration method would then be dropped.

Research Program

The setting is a holder with a private dataset split into train and test. The holder trains a teacher on the private train split only. It then releases a surrogate made of three parts. The first is a set of inputs, drawn from one of three sources: a public proxy corpus, synthetic inputs generated from the teacher, or differentially private synthetic text. The second is labels produced by the teacher, optionally privatized. Privatizing means adding calibrated noise so that no single record can be inferred, which is the guarantee formally called differential privacy. The third is a surrogate test split.

Outsiders train candidate recipes on the surrogate, meaning combinations of architecture, augmentation, and regularization. They then compare the recipes. Ground-truth fidelity is the Kendall rank correlation between recipe scores on the surrogate and scores when the same recipes are trained and tested on the real private data. We also report two finer measures. One is top-1 agreement. The other is the flip rate on pairs whose real gap exceeds seed-to-seed noise, so that ties caused by noise are not counted as errors.

The method ingredient is rank calibration. The holder runs K reference recipes, with K around 8, on the real data. It then searches over surrogate design choices: which subset of proxy inputs to keep, the label temperature, and hard versus soft labels. It picks the configuration whose induced ranking best matches the reference ranking. Only the K scalar scores touch private data during this search. They are released or used with added noise, so their privacy cost is small and countable.

A previously awkward feature of data-free methods becomes harmless here. Data Impressions require the teacher's weights (Mining Data Impressions From Deep Models), and CAKE requires gradients with respect to the teacher's inputs. The holder owns the teacher, so it has both, and these methods become legitimate input generators.

The soft-label risk comes from DD-Ranking. That paper showed that random real images given teacher soft labels beat distilled datasets: the SRe2L method scored −1.5% to −15.6% improvement over random. The implication is that soft labels carry most of the measured signal. We test whether that same strength compresses the score spread across recipes and lowers rank correlation. DD-Ranking's label-robust and augmentation-robust scores serve as diagnostics.

A second risk comes from Data Laundering. That paper showed that teacher logits on unrelated data can transfer test-set knowledge to a student: 73.94% on GPQA Diamond with a 2-layer BERT. We therefore include a deliberate check. A teacher contaminated with the private test split is run through the pipeline. A useful surrogate protocol should expose this, because contaminated rankings should diverge from rankings produced by a clean teacher.

Baselines

The naive reference is to rank recipes on a public dataset from the same domain using its own labels, with no teacher involved. This is what researchers do today when they cannot access private data. Beating it means a clearly higher rank correlation, with a confidence interval across resampled recipe pools that excludes the naive value.

The strongest simple competitor is teacher-labeled random public proxy data. This is the self-learning plus distillation setup of RONA and the "random plus soft labels" baseline that DD-Ranking found so hard to beat. We run it with the same proxy pool, the same teacher, and the same label budget, and give it the same tuning budget for temperature as the calibrated method.

For privatized labels we use reverse k-nearest-neighbor labeling from Fine-grained Private KD at matched privacy budget ε. Its reported accuracy is 82.1% on CIFAR-10 at ε = 1.0. No reported result found on Lacuna for its rank fidelity. For the setting with no proxy data we use Data Impressions and CAKE as input generators. For text we use the DP-Gen and AUG-PE synthetic corpora from SynBench, generated with the same LLaMA-3 sizes and ε ≤ 4. That paper reports utility collapse in this regime but no rank-fidelity figure. No reported result found on Lacuna for ranking with DP synthetic text.

Ablations isolate the key ingredients. We compare calibration on versus off with everything else fixed. We compare hard, tempered, and full soft labels. We vary K to see how many private reference runs calibration needs. Calibration counts as beating the strongest competitor only if it wins on held-out recipes. Wins on the reference recipes themselves do not count.

Why not just…?

Why not just run an evaluation server where outsiders submit code and the holder runs it on private data? This is the real verification mechanism, and we treat it as complementary. It does not support the iterative development loop of ablations, error analysis, and hundreds of cheap runs. Each query also spends compute and potentially privacy. In our design the surrogate is the development tool and a server is the final check. We will report how many server calls the surrogate saves at fixed top-1 agreement.

Why not just de-identify and release, as MIMIC-III did? MIMIC-III shows that this works for some clinical records. Biometric data cannot be de-identified without destroying the signal, as with the ID photos in DocFace. Commercial logs are withheld as trade assets, as with the email corpus in Domain Adaptation for Enterprise Email Search. Our method targets those cases.

Why not just release teacher-labeled random public data? This is the serious threat. DD-Ranking suggests it may be very strong. It is our main baseline, and if it matches the calibrated surrogate on held-out recipes, we will say so and drop calibration. Our prediction that soft labels flatten score spread gives a specific way it could lose. The prediction is falsifiable by measuring score spread directly.

Why not just release DP synthetic data? SynBench reports that under ε ≤ 4, synthetic text fidelity measured by MAUVE falls near zero. It also finds that apparent success often reflects pre-training memorization. We include it as a baseline, and we use non-memorized splits so that this illusion does not inflate its numbers.

Proposed Experiments

Images come first. CIFAR-10 plays the private dataset, with ground truth computed on real data. Proxy pools are drawn from mismatched public image sets, SVHN-like digits are used to stress domain shift, and the data-free generators cover the no-proxy case. The recipe pool has about 40 recipes spanning ResNets, MLPs, and ViTs, with varied augmentation and regularization, each run over several seeds.

Clinical text comes second. The i2b2 and MedNLI tasks from Lightweight Transformers for Clinical NLP play private datasets. The recipe pool is that paper's compact clinical students, from 15M to 65M parameters, under different fine-tuning recipes.

The decisive checks follow from the claims. Calibrated surrogates must beat teacher-labeled random proxy data on held-out recipes. Hard or tempered labels must give higher rank correlation than full soft labels at equal or lower student accuracy. The contaminated-teacher condition must produce measurably distorted rankings. If none of these hold, the honest conclusion is that simple teacher relabeling suffices. That would still be a first quantitative answer to whether distilled stand-ins can support claims made on private data.