Randomized Compiling for Preference Collection

Language models are aligned using pairwise human judgments ("response A is better than B"). A large share of these judgments are wrong. On Anthropic-HH, a quarter of re-examined pairs had the "rejected" answer as the better one, and annotators agreed with each other only moderately (How Reliable Is Human Feedback For Aligning Large Language Models?). The best-founded fix, robust DPO, corrects the training loss without bias, but only if every label is flipped independently with one known probability (Provably Robust DPO). Real annotation noise does not behave that way. Annotators favor one screen position, tire over a session, and apply shifting thresholds; threshold variance alone accounts for 37% of low-agreement pairs in the Yeh et al. audit. The noise has memory, it depends on presentation, and it is tangled up with which prompts happen to land late in a tired session.

Quantum computing faced the same mismatch and solved it with randomized compiling. The technique inserts random, known gate changes ("frames") into a circuit and compiles the corrections back in. Structured, history-dependent errors then average into simple independent random flips that standard models describe well. On a superconducting processor, a goodness-of-fit violation score dropped from about 535 to about 25 under this randomization (Benchmarking quantum logic operations relative to thresholds for fault tolerance). We propose to apply the same recipe to annotation itself. Each comparison is shown under a random known frame: which response appears on which side, which slot in the session it occupies, and which annotator sees it. The frame is undone when labels are recorded. A small number of comparisons with answers known by construction are mixed in so the remaining flip rate can be measured. The goal is preference data whose noise actually satisfies the assumptions of robust DPO, together with an estimate of that noise that does not require a clean validation set.

Core Claim

Randomizing the presentation frame does not reduce how much noise there is. Twirling cannot remove noise energy, it only reshapes it. What it does is change the form of the noise. Two properties make that useful. First, randomly swapping which response appears on the left turns an asymmetric position bias into a symmetric flip whose probability does not depend on which response is truly better. Second, randomly assigning items to session slots and annotators makes fatigue and drift independent of prompt content. Together these give an approximately content-independent flip process with a single average rate. That is exactly the condition under which the robust DPO estimator is unbiased.

The second ingredient borrows from mirror-circuit benchmarking. Mirror circuits are built so that the ideal output is known in advance, which lets the error rate be measured without simulating the circuit (Scalable randomized benchmarking of quantum computers using mirror circuits). Our analogue is "mirror probes." These are comparisons whose answer is fixed by construction: the same pair shown twice with sides swapped, or a response set against a deliberately degraded copy of itself. Probes are scattered through the twirled sessions and estimate the flip rate as a function of session position and annotator. That estimate replaces the hand-tuned noise rate in robust DPO and the clean validation set that PerpCorrect needs.

Research Program

The first component is a collection protocol. It defines the frame group (side swap, session-slot permutation, annotator assignment, and optionally rubric paraphrase) and a probe rate of roughly 5–10% of the budget. Probes are charged against the same annotation budget as real pairs.

The second component is a noise-model test, analogous to the violation score from gate set tomography. We fit two models to the probe and repeated-pair responses. One is an independent symmetric flip model. The other is a richer model with position, session-time, and annotator effects. The likelihood-ratio statistic between them, expressed in standard deviations, measures how badly the simple model fails. We predict this statistic drops sharply under twirling, as the quantum statistic did.

For the leftover misfit we borrow wildcard error modeling from the same quantum paper. We find the smallest extra flip margin $w$ that makes the independent-flip model statistically consistent with the data, and train robust DPO at $\hat\epsilon + w$ rather than at $\hat\epsilon$, where $\hat\epsilon$ is the probe-estimated rate. This gives a conservative, data-derived noise setting.

The third component addresses memory. Fatigue is a hidden annotator state that persists across items. The quantum literature warns that hidden memory produces non-exponential, sometimes non-monotone accuracy curves over sequence depth (Towards a general framework of Randomized Benchmarking incorporating non-Markovian Noise). We therefore fit probe accuracy against session position with a small hidden-state model rather than assuming a decay. We then compare two ways of handling it. The fully twirled version uses one averaged rate. The frame-resolved version weights each label by $\hat\epsilon(t)$, its slot's estimated rate:

$$\mathcal{L} = \frac{(1-\hat\epsilon(t))\,\mathcal{L}_{\text{DPO}}(y_w \succ y_l) - \hat\epsilon(t)\,\mathcal{L}_{\text{DPO}}(y_l \succ y_w)}{1 - 2\hat\epsilon(t)}.$$

Here $\mathcal{L}_{\text{DPO}}$ is the standard DPO loss. This is robust DPO with the global rate replaced by a slot-specific one. Because slot is randomized, $\hat\epsilon(t)$ is uncorrelated with content, so the per-item estimator stays unbiased.

Our scope claim is limited. Twirling cannot fix noise that is tied to the content of a pair: genuine subjectivity, or cases where both responses are bad. These play the role that state-preparation and measurement errors play in randomized compiling, which the technique also leaves untouched. We will report how much of the total noise falls in this content-tied part.

Baselines

The naive reference is the standard pipeline: fixed side order, items labeled in whatever order they arrive, and vanilla DPO. On Golden HH with 40% independent synthetic noise, PerpCorrect reports vanilla DPO collapsing to 53.15% reward accuracy (Perplexity-aware Correction for Robust Alignment with Noisy Preferences).

The strongest loss-side competitor is robust DPO trained on naively collected data, with its noise rate tuned by grid search on a held-out clean set. Under 40% independent noise on IMDb it reports mean reward 0.99 against 0.88 for DPO, and it is the method our protocol is designed to make valid. For a fair comparison it gets the same models (Pythia-2.8B and Llama-2-7b), the same number of training steps, and the same tuning grid. It also gets the clean labels our method does not use.

The strongest data-side competitor is PerpCorrect. It flags flipped labels by the gap in model perplexity between the two responses, using 50 clean samples. On Golden HH at 40% independent noise it reports 94.92% reward accuracy, which is the number to beat. It will also be run on top of twirled data, since the two methods can be stacked.

NS-DPO down-weights older labels exponentially to handle shifting preferences (Right Now, Wrong Then). It is the natural competitor for time-structured noise; it reports reward accuracy holding above 60–70% where DPO falls to about 50–55% at change points. β-DPO adapts the DPO strength per batch and filters outliers (β-DPO). It reports 57.07% against 51.51% GPT-4 win rate on Pythia-2.8B, and we include it as a second loss-side method.

Two ablations isolate our ingredients. "Shuffle-only" randomizes side and order but uses no probes and trains robust DPO with a tuned rate. "Probes-only" adds probes to naive collection. Beating any of these means higher reward accuracy on the gold labels and a higher GPT-4 win rate at equal total annotation budget. None of the published numbers above were obtained under structured noise; no reported result found on Lacuna for any of these methods under position-biased or fatigue-correlated preference noise.

Why not just…?

Why not just randomize A/B order and shuffle items, as careful crowdsourcing already does? This is the real threat. Our shuffle-only ablation is exactly this. If shuffling plus robust DPO with a rate tuned on clean data matches our method, then frame randomization alone adds nothing. What remains would be the probe-based rate estimate and the misfit test, which we would then present as a way to drop the clean validation set rather than as an accuracy gain. The experiment settles this directly.

Why not just use label smoothing? Smoothing the targets toward the noise rate (conservative DPO) is provably biased. The robust DPO paper shows it is beaten at high noise. It also still needs a noise rate, which is the quantity our probes supply.

Why not just collect several labels per pair and take a majority vote? At equal budget, redundant labeling buys fewer distinct pairs. When annotators share biases or fatigue together, as the 0.46 agreement score suggests, a vote averages correlated errors and fixes less than it should. This alternative could still be competitive, so it will be run as an equal-budget baseline.

Why not just discount old labels with NS-DPO? NS-DPO treats change over time as a real change in what people prefer. Annotator fatigue is noise around a fixed truth, so discounting throws away valid data. Probes have fixed answers, so their accuracy tracks fatigue independently of any real preference drift, and the two cases can be told apart. We test this with a simulated arm that contains both.

Proposed Experiments

Testing this needs gold labels and controllable structured noise, so the first stage is simulated annotation on Golden HH and UltraFeedback. Simulated annotators carry a persistent fatigue state, a side bias, and annotator-specific thresholds, and their error rates are matched to the Yeh et al. audit. A second annotator pool uses LLM judges, which show real, uninjected position bias. Fixed annotation budgets are spent under naive, shuffle-only, and fully twirled collection. Models are then trained with every baseline and evaluated by gold reward accuracy and GPT-4 win rate. IMDb provides a cheap sanity check, comparing against robust DPO's reported 0.99.

The decisive checks follow from the claims. First, the misfit statistic should fall sharply under twirling while the average flip rate stays roughly the same. Second, twirled robust DPO should beat naive-collection robust DPO even though the naive version gets a tuned rate. Third, probe-estimated rates should fall within a few points of the true injected rate without any clean labels. Fourth, the gain should vanish when only content-tied noise is injected; if it does not, our mechanism is not the explanation.

A small pilot with human annotators on 2,000 HH pairs would check whether real fatigue curves are non-monotone, and whether twirling reduces real misfit as it does in simulation. JustQ (JustQ) used randomized compiling to make hardware noise predictable for an optimizer, but it treated its labels as clean. Our work moves the randomization to the point where the labels are produced.