Cloaked Twins: Public Benchmark Copies That Do Not Teach Their Own Answers

Benchmark curators face a bad choice. If they publish test questions as plain text, web crawlers pick them up, models memorize them, and leaderboard scores inflate. If they hide the questions behind a password or a private server, the community loses the ability to read, audit, and analyze the items, and a single person who re-uploads the decrypted file undoes the protection. We propose a third option: publish a deliberately altered "cloaked twin" of each test item. The twin reads as the same problem to a person or to a model that has never seen it. It is built so that a model trained on the twin gains nothing when later tested on the original, untouched item, which stays in isolated evaluation. The alteration uses "unlearnable" rewrites. These are small, meaning-preserving edits planted so that a model trained on the text latches onto an easy shortcut in the edits instead of the answer.

The open question is the one that adversarial-transferability research has long studied in a different setting. Adversarial transferability is the observation that inputs crafted to fool one model often fool other, unseen models. Here the curator can optimize the cloak only against a few small local models, called surrogates. The training pipelines that will eventually scrape the twin are unknown: other model families, other tokenizers, other scales, and pretraining rather than fine-tuning. The proposal is to build cloaked twins and to measure, under a careful transfer protocol, whether their "no contamination gain" property survives across that gap.

Core Claim

Contamination mitigations that edit the evaluated item hit a known trade-off. The Emperor's New Clothes in Benchmarking? tested 20 strategies. It found that surface edits keep fidelity above 0.9 but give almost no contamination resistance, while aggressive multiple perturbations push MMLU fidelity down to 0.686. Our design avoids this trade-off by construction, because the item that is scored is never edited. Only the copy that circulates publicly is edited, and its job is to absorb whatever training happens on it. The claim is that a cloak optimized against an ensemble of surrogates, with an objective that directly targets the clean-item gain, can drive the contamination gain on clean items to near zero for held-out model families. Simple decoys such as resampled numbers or permuted answer options will not, because they block memorization of answer strings but still teach the solution pattern.

Define the contamination gain for a release format $r$ as $G_r = \text{acc}(M_r, D) - \text{acc}(M_\varnothing, D)$. Here $D$ is the clean test set, $M_r$ is a model trained on a corpus containing release $r$, and $M_\varnothing$ is the same model trained without it. Our main metric is resistance $R = 1 - G_{\text{twin}}/G_{\text{clean}}$, where $G_{\text{clean}}$ is the gain from leaking the plain items. $R=1$ means the twin teaches nothing about the test, and $R<0$ is not desirable. The second constraint is solvability fidelity: models that have never seen either copy should solve the twin and the original at matching rates and in the same model ranking.

How This Relates to Prior Work

The method comes from unlearnable-example research. Transferable Unlearnable Examples plants shortcut perturbations that drop CIFAR-10 supervised accuracy from 93.79% to about 10%, and they transfer across training paradigms and datasets. TextCloak carries this to text. It uses a reinforcement-learning rewriter trained in a two-level loop: the inner level trains a surrogate on the rewritten text, the outer level updates the rewriter. The rewrites cause an average 12.4% drop on ARC-Challenge, MMLU-Pro, MATH and other tasks at 0.95 SBERT similarity. Both works protect private data from unauthorized fine-tuning, and both measure lost general skill. We reuse TextCloak's rewriter machinery but change the target in three ways. We aim at zero gain on specific held-out items rather than degraded general skill. We test exposure through pretraining-style corpus mixing rather than fine-tuning. We add a solvability constraint so the twin remains a usable stand-in for readers. TextCloak's reported cost of about 40 seconds per batch on an A100 is prohibitive for large corpora but trivial for a benchmark of a few thousand items.

The motivation comes from the distribution literature. Stop Uploading Test Data in Plain Text recommends encrypted archives but calls them a speed bump that breaks after a single leak. It also reports that 4.2 external models cite an AI2 leaderboard dataset for every one that submits officially, which shows that readable copies get used. Private-server benchmarks such as GSM1k and remote sandboxes such as TIRA keep the clean set isolated, and we assume that kind of isolation for the scored copy. The cloaked twin is meant to be the readable artifact that people circulate in place of the clean one. The size of the problem is documented: GSM1k shows drops of up to 8% against GSM8k for some model families, and a Text-to-SQL study shows drops of up to 20% on unindexed data.

Two lines of work constrain the design. First, Reliable Evaluation of Adversarial Transferability shows that transfer is overstated when testing stays within one architecture family and that it collapses across families. This cuts both ways for us. Cloaks may fail to transfer across tokenizers and families, and any apparent success must be checked against same-family controls. The same concern applies to encrypted-model transfer. That paper reports that attack success drops to 24.37% against FFX-encrypted ResNet18, and part of that drop could reflect architecture divergence rather than the secret key. We therefore separate surrogate–target similarity from the cloak's own effect, using seed-only, tokenizer-only, and family-level differences. Second, Detecting Benchmark Contamination Through Watermarking rephrases benchmarks with a watermarked model so that training on them leaves a detectable statistical trace. That approach detects contamination but does not prevent it. Its experimental protocol supplies our testbed: 1B models trained from scratch on 10B tokens with controlled contamination, across Llama-3 and Gemma tokenizers. A side benefit we will test is that a model which has learned the cloak's shortcut should respond to it, which would give detection for free.

Research Program

We will build a twin generator for two item types. The first is multiple-choice items (ARC-Easy, ARC-Challenge, MMLU), where the shortcut can tie lexical cues in the stem to the answer option. The second is free-form arithmetic (GSM8K), where the shortcut must coexist with a worked solution. The rewriter follows TextCloak's group-relative policy optimization. Its reward has three parts. The first is the negative clean-item gain measured on surrogates after brief training on the twins mixed into web text drawn from FineWeb. The second is semantic similarity together with a perplexity ceiling, since Is LLM-as-a-Judge Robust? shows that high-perplexity adversarial strings are flagged at F1 0.7–0.8 and the twin must not look strange. The third is solvability parity, meaning matched accuracy of unexposed surrogates on twin and original. The surrogate ensemble spans two tokenizers and two sizes, and test-time targets come from families held out of the ensemble.

The analysis rests on separating two channels of contamination. One is answer-string memorization; the other is learning the solution pattern for the specific item. Naive decoys block only the first, and the optimized cloak is meant to block both. For each release format we will measure $G_r$ split by item and relate it to the target model's likelihood on the original item, following GSM1k's finding of a Spearman $r^2$ of 0.36 between memorization likelihood and the performance gap.

Baselines

The do-nothing reference is releasing the clean items in plain text and training on them, which defines $G_{\text{clean}}$ and $R=0$. The opposite reference is full withholding, as in GSM1k or an encrypted archive that never leaks, which gives $R=1$ by definition but provides no readable copy. Each method is judged by where it falls between these two.

The strongest simple competitor is a naive decoy twin. For arithmetic, it resamples the numbers so that every answer changes. For multiple choice, it permutes the options and rewrites the stem with a plain LLM paraphrase. It costs nothing and directly breaks answer memorization, which is why it is the main threat. The strongest existing optimized method is TextCloak applied off the shelf with its original objective of degrading general skill. Watermark rephrasing from the watermarking paper serves as the detection-only baseline. That paper reports a $p$-value of $10^{-5}$ when contamination inflates scores by 5%, and $p<10^{-12}$ after 16 exposures on ARC-Easy. We match it on the detection axis and compare on resistance, where it is not designed to help.

All conditions use the same 1B and larger targets, the same 10B-token mixtures, the same contamination rates of 1, 4 and 16 exposures, and the same item sets. Optimized methods receive equal surrogate compute and tuning budgets. No reported result was found on Lacuna for clean-item contamination gain after training on any rewritten copy, so every baseline number is produced inside this study.

Beating the competition means three things on held-out families: $G_{\text{twin}}$ within seed noise of zero, a gain clearly below the 5% inflation level the watermarking paper treats as meaningful contamination, and naive decoys retaining a substantial fraction of $G_{\text{clean}}$. All of this must hold while solvability rank correlation with the originals stays high.

Two ablations isolate the key ingredient. The first removes the clean-gain term from the reward, leaving a semantics-preserving rewrite with no optimization against learning. The second compares a single-surrogate cloak with the ensemble cloak, which tests whether transfer, rather than rewriting per se, does the work.

Why not just…?

Why not just withhold the test set or ship it encrypted? Isolation is the right home for the scored copy, and we keep it there. But the "Stop Uploading" authors themselves describe encrypted archives as defeated by one re-upload, and the 4.2-to-1 shadow-usage ratio they report shows that people want and spread readable copies. The twin is meant to satisfy that demand harmlessly. If the twin offers no advantage over withholding, it still adds value only if it reduces leaks of the clean copy, a social effect this study cannot measure, and we will say so.

Why not just publish a decoy with changed numbers or shuffled options? This is the real threat, and our naive-decoy baseline tests it directly. The Emperor's New Clothes result, that contaminated models see through surface edits, points in the reverse direction from our setting, so it suggests rather than proves that training on near-copies still transfers the solution pattern. If number resampling achieves resistance comparable to the optimized cloak on held-out families, we will drop the optimization and recommend the simple decoy. That would itself be a useful, previously unmeasured result.

Why not just paraphrase the released copy? Paraphrase is the cheapest decoy. The same 20-strategy study found it gives almost no contamination resistance when used to edit the evaluated items. Our paraphrase-only ablation tests whether it also fails when it edits only the released copy.

Why not just watermark and detect? Detection identifies offenders after scores are already inflated, and it relies on leaderboards choosing to act on a $p$-value. Prevention keeps the score meaningful without anyone's cooperation. We report both, and we check whether the cloak's shortcut provides comparable detection power as a side effect.

Decisive Checks

The idea fails if any of three outcomes holds. The first is that cloaks reach $R$ near 1 on surrogate families but fall to near the naive decoy on held-out tokenizers and families, the cross-family collapse the transfer-evaluation literature predicts. The second is that an attacker who simply paraphrases the twin before training recovers most of $G_{\text{clean}}$, so the cloak washes out. The third is that solvability parity breaks, meaning unexposed models rank differently on twins than on originals.

The study is structured so that each failure is informative. In the first case, we learn how far unlearnability transfers at pretraining scale. In the second, we learn that item-specific shortcuts are fragile, unlike the class-wise shortcuts in vision. In the third, we learn the true cost in fidelity of a twin that cannot be learned from.