Judges That Know "Worse Than" Is the Transpose of "Better Than"

When a language model is asked to judge which of two answers is better, it often gives the wrong answer to the mirror-image question. It can contradict itself when the answers are presented in the other order, or when it is asked which answer is worse. The research on the "reversal curse" in language models describes a structurally similar failure. A model trained on "Alice's husband is Bob" cannot answer "who is Bob's wife?", because nothing forces its internal representation of a relation to be tied to the representation of the inverse relation. Our hypothesis is that judge inconsistency is partly the same failure applied to the comparative relation "better than". If so, it can be fixed the way synthetic reversal failures have been fixed: by giving the relation a form in which inversion is a transpose.

Concretely, we propose a judge that reads both answers together, as generative judges do. It then outputs its preference through a small comparison layer with two parts. The first is a per-answer quality score, which yields a ranking with no cycles. The second is a "skew-symmetric" interaction term, a pairwise term that flips sign exactly when the two answers swap roles. Under this design, order consistency and "better/worse" consistency hold by design rather than being taught through augmented data. Every cycle the judge produces must pass through one inspectable term, so we can measure it, penalize it, or check whether it tracks real human disagreement.

Core Claim

Bilinear representation mitigates reversal curse and enables consistent model editing shows that the reversal curse eases when relations are encoded in middle layers as a bilinear form $s^\top M o$, because the inverse relation is then simply $M^\top$. That paper only studied synthetic family trees. We carry the idea over to pairwise judging. Let $h_A$ and $h_B$ be pooled middle-layer states of the judge over the spans of answers $A$ and $B$, read in a single pass with both answers in context. Let $u$ be a learned quality direction, and let $K$ be a learned low-rank matrix. The preference logit is

$$\ell(A \succ B) = u^\top (h_A - h_B) + h_A^\top (K - K^\top)\, h_B .$$

The matrix $K - K^\top$ is skew-symmetric, so swapping $A$ and $B$ negates both terms. The query "is $B$ worse than $A$?" is answered with the transposed form, so it returns exactly $\ell(A \succ B)$. The first term is a difference of scores and so can never produce a cycle like $A \succ B \succ C \succ A$. Only the skew term can. A skew-symmetric form of rank two can represent a rock-paper-scissors cycle, so this term is where any intransitivity in the judge has to live. Our claim has three parts. First, off-the-shelf judges encode "better" and "worse" through operators that are not transposes of each other, and the size of this mismatch predicts how often they fail the better/worse check. Second, constraining the readout as above beats data-augmentation fixes on consistency, and does not lose accuracy compared with an unconstrained comparative head. Third, the size of the skew term on a given pair is a useful signal: it should rise on pairs that are close in quality or that trade off different criteria, which is where humans also disagree.

Relation to Prior Work

The motivating evidence comes from work that measures judge inconsistency without changing how the judge represents comparisons. Judging the Judges measures order-swap behavior in 15 judges. It finds the bias is systematic rather than random: repetition stability is above 0.95, while position consistency collapses when the two answers are close in quality. Investigating Non-Transitivity in LLM-as-a-Judge finds that preference cycles peak in this same close-quality regime. Its remedy is a change at evaluation time: round-robin and Swiss-style tournaments scored with a Bradley-Terry model, which assigns each answer one latent strength. Diagnosing LLM Judge Reliability shows that 33–67% of SummEval documents contain at least one preference cycle. It also shows that post-hoc graph repair does not recover human rankings, which it reads as genuine confusion by the judge. All three studies diagnose or aggregate around the judge. None of them changes the comparison the judge computes, which is what we do.

The closest work is Aligning with Logic. It defines transitivity, commutativity (order invariance) and negation invariance (better/worse consistency). Its REPAIR method fits a global ranking, generates a conflict-free set of comparisons from it, and adds negated queries to the fine-tuning data. This raised negation invariance on Summarize-from-Feedback from 63.8% to 87.9% for Llama-3-8B. The paper also reports a "forgetting effect", in which the negation data degrades other consistency properties. REPAIR teaches consistency through data. We build it into the readout and leave the training data unchanged. That is why the forgetting effect is the specific failure our design should avoid. REPAIR also forces all comparisons onto one linear order, which removes any cycles that humans themselves produce. Our design keeps them in a separate, regularized term.

The method side borrows from inverse-relation work. Reciprocal-edge augmentation in knowledge-graph models (Lacroix et al.), Identity Bridge and Deductive Closure Training all fix reversal by adding inverse statements to the data. Identity Bridge breaks down on entities longer than a few tokens: accuracy is near 100% on single-token entities and around 10% on three-token entities. Judge inputs are hundreds of tokens long, so the data route looks especially unpromising here. This is a constraint that pushes us toward a structural fix. Reciprocal Perspective Calibration explicitly lists "explicit consistency loss functions" for inverse relations as open. Our skew-symmetric parameterization is a stricter answer to that problem than a loss term. Finally, Learning LLM-as-a-Judge for Preference Alignment shows that comparative generative judges (Con-J) beat scalar reward models. This is why we keep the cross-attending pass instead of falling back to scoring each answer on its own.

Research Program

The work has three stages.

Stage 1: probe existing judges. We probe Llama-3-8B and a second open judge. Using paired "which is better" and "which is worse" prompts over the same answer pairs, we fit separate bilinear probes $M_{\text{better}}$ and $M_{\text{worse}}$ on pooled answer-span states at each layer. We then measure how far $M_{\text{worse}}$ is from $M_{\text{better}}^\top$. The prediction is that a layer band with good probe accuracy exists, as in the synthetic bilinear study, and that within it the transpose mismatch correlates with negation-invariance failures across pairs.

Stage 2: train the comparison head. We attach the head to that layer band and train it with LoRA on the backbone. Pooling over answer spans removes a direct role for position. Position can still leak into $h_A$ itself, though, so we add a training-only term. It pulls an answer's pooled state toward the state it receives in the swapped prompt. Inference stays a single pass, and an ablation tests whether this term is needed. We also add a penalty on the norm of the skew matrix, weighted by $\lambda$. Sweeping $\lambda$ traces a frontier between how well the judge fits human labels and how many cycles it produces.

Stage 3: analyze the skew term. We ask whether pairs where the skew term is large are the pairs where human raters disagree, or where SummEval's per-criterion ratings point in opposite directions. If they are, the cycles are informative rather than noise.

Baselines

The naive reference point is the base judge prompted zero-shot in one order. It reports the raw position and negation consistency we start from. On Summarize-from-Feedback, its negation invariance is 63.8% for Llama-3-8B according to Aligning with Logic.

The strongest consistency baseline is REPAIR, from the same paper. We reimplement it on the same Llama-3-8B backbone, with the same human-labeled pairs as seed data and the same LoRA rank and step budget. Its reported 87.9% negation invariance on Summarize-from-Feedback is the bar. Our design achieves negation invariance by construction, so on that metric alone matching REPAIR would be trivial. To count as beating REPAIR, we must match or exceed its accuracy against human preference labels and its position consistency. We must also show no drop in transitivity, measured by the percentage of non-transitive triples, which is where REPAIR's forgetting effect appears.

The strongest accuracy baseline is a Con-J-style generative judge trained with rationales on the same data. Its reported Infinity-Preference accuracy is 81.0%, against 75.0% for GPT-4o (Learning LLM-as-a-Judge for Preference Alignment). We expect a small loss in accuracy against it. Our success criterion is to stay within one point of its accuracy while substantially improving position consistency. The most consistent judges in Judging the Judges, Claude-3.5-Sonnet and GPT-4, reach about 0.82.

A Bradley-Terry scalar reward model on the same backbone is the structural baseline. It scores each answer independently and is antisymmetric and transitive by construction.

The key ablations vary the interaction matrix while holding everything else fixed:

  • an unconstrained bilinear matrix, which keeps the comparative pass but drops the skew constraint;
  • a zero matrix, which keeps only the quality-difference term computed with both answers in context;
  • the full model without the swap-tying term.

The gap between the unconstrained and skew-symmetric heads isolates the contribution of the transpose structure.

Why not just…?

Why not just query both orders and average? Swapping and averaging makes the order-swap check pass, but it doubles inference cost. It also does nothing about cycles across triples. In the close-quality regime, where Judging the Judges shows the bias is systematic, averaging mostly turns confident wrong answers into ties. This is a real competitor on accuracy, so we compare against it at matched compute: our single pass against its two passes, and also against its two passes using a judge twice as cheap. We report accuracy on near-tie pairs separately. If swap-averaging matches us at equal compute, our contribution narrows to better/worse consistency and the cycle diagnostic.

Why not just use a scalar reward model, which is consistent for free? This is the most serious threat, because a Bradley-Terry model already has every symmetry we enforce. The evidence against it is that comparative judges beat scalar ones (Learning LLM-as-a-Judge for Preference Alignment). That comparison, however, mixed several differences between the two systems. Our zero-matrix ablation, which computes scores with both answers in context, and the independent Bradley-Terry model separate the benefit of reading both answers together from the benefit of the skew term. If the independent Bradley-Terry model matches our full model on human agreement, we will report that the comparative pass is not worth its cost for these tasks.

Why not just fine-tune on swapped and negated data, as REPAIR does? That is exactly the REPAIR baseline. Its own reported forgetting effect, together with Identity Bridge's breakdown on long inputs, are the reasons we expect data-only inversion to transfer poorly to long answers. The head-to-head comparison above settles the question.

Why not just ask for chain-of-thought reasoning? Aligning with Logic reports that chain-of-thought can lower transitivity by adding variance. We include it as a prompted baseline for completeness and do not expect it to compete on consistency.

Proposed Experiments

Data. Training pairs come from Summarize-from-Feedback and from SummEval. For SummEval we build pairs from the expert Likert ratings, which have Krippendorff's alpha 0.71 (SummEval). Held-out evaluation uses MT-Bench pairwise human judgments and Infinity-Preference. For ranking whole models we run the judge in round-robin on Arena-Hard-style prompts and compare against Chatbot Arena. There, tournament scoring with an off-the-shelf judge reached 96.4% Spearman correlation (Investigating Non-Transitivity in LLM-as-a-Judge). We test whether our judge in the cheaper fixed-baseline setup closes the gap to that tournament number. The paper reports that only 20% of models keep their rank when the baseline changes, and this rank stability is our second ranking metric.

Metrics. We report accuracy against human labels, position consistency, negation invariance, the percentage of non-transitive triples, and the soft non-transitivity score from the same paper. Each is also broken down by bins of quality gap between the two answers.

First decisive check: the probing prediction. If the transpose mismatch between the better and worse probes does not correlate with better/worse failures, the mechanistic motivation fails. In that case the head becomes an engineering choice, and we would say so.

Second decisive check: the near-tie bins. The skew-symmetric head must improve position consistency where judges fail most, which is near-tie pairs, while costing at most one point of overall accuracy against the unconstrained head and the Con-J judge.

Third decisive check: the cycle analysis. On SummEval documents with cycles, the skew-term magnitude should predict pairs where expert ratings disagree or trade off across criteria. We measure this with AUC, against a predictor based only on the quality gap. If it does, the cycles that Diagnosing LLM Judge Reliability found irreparable after the fact can at least be located and flagged during judging. If it does not, the right setting is a large $\lambda$, and the method reduces to a cheaper, in-context Bradley-Terry judge.