Language-model alignment increasingly trains a policy against a rubric: a list of criteria such as "states the correct dosage" or "asks about allergies," each graded by a larger model acting as judge. The strongest recent result, Alternating Reinforcement Learning with Contextual Rubric Rewards (ARL-RR), shows it helps to optimize one family of criteria at a time. Averaging all criteria into one number shrinks the reward's variance, which weakens the learning signal and invites reward hacking. The authors also flag the weak point of their own method. When the judge is noisy or contradicts itself, alternation can amplify those errors from one training stage to the next, because each stage trusts a single criterion family completely.
This proposal asks the judge two separate questions per criterion instead of one: how much evidence says the response satisfies it, and how much evidence says it violates it. That split separates two failure cases a single score merges. In the first, the judge sees little evidence either way, which is ignorance. In the second, it sees strong evidence both ways, which is conflict. A mathematical object from fuzzy decision theory fits this pair exactly: the q-rung orthopair fuzzy set, a (support, against) pair allowed to be jointly large, with a tunable exponent $q$ that controls how large. We use the pair to decide which rubric items to trust and when to optimize each criterion family. The alternation then runs over two coupled parts. One is the policy. The other is a small set of reward-side parameters, re-fit in closed form from fresh rollouts.
Core Claim
Our hypothesis concerns noise in rubric-based RL: it is not all alike. Items where the judge is conflicted are the ones most likely to create spurious uphill directions that the policy can exploit, such as hedged, verbose answers that satisfy the letter of a criterion while also violating it. Items where the judge is merely ignorant are mostly harmless low-signal noise. If this is right, masking or deferring conflicted items should beat any method that uses a single confidence number, and the gap should widen as judge noise grows. If conflict and ignorance behave the same, the fuzzy-set machinery reduces to ordinary confidence weighting and should be dropped. The experiments are built to tell these apart.
The representation
For a rubric item, the judge is queried independently for a support degree $\mu \in [0,1]$ and an against degree $\nu \in [0,1]$, for example as the probability of "yes" to "is this criterion satisfied?" and to "is this criterion violated?" in separate prompts. Because the queries are independent, pairs like $(0.8, 0.7)$ will occur.
Classical intuitionistic fuzzy sets require $\mu+\nu\le 1$ and would reject such a pair. Pythagorean fuzzy sets, as used in Pythagorean fuzzy time series model based on Pythagorean fuzzy c-means and improved Markov weighted in the prediction of the new COVID-19 cases, relax this to $\mu^2+\nu^2\le 1$ and still reject it. The q-rung version requires $\mu^q+\nu^q\le 1$ and admits the pair once $q\ge 3$. We define the item's signed score $s$ and its conflict $c$ as
$$s = \mu^q - \nu^q, \qquad c = \min(\mu,\nu)^q,$$
where $s$ plays the role of the usual score function in this literature and $c$ is large only when both kinds of evidence are strong. The remaining room, $1-\mu^q-\nu^q$, is the conventional hesitancy and captures ignorance.
A known open problem in this literature is that $q$ is chosen by hand and checked only afterwards by sensitivity analysis, for instance in Generalized q-ROF TODIM Decision-Making Method Considering Attribute Correlation. Here $q$ is instead fit from data: for each criterion family it is the smallest value that makes a target fraction of observed judge pairs admissible. That fit is the reward-side update described below.
Research Program
Training follows ARL-RR's five criterion families: Accuracy, Completeness, Instruction Following, Context Awareness and Communication Quality. The modification is a two-block alternation.
In the policy block, the reward-side parameters are held fixed. The policy takes a round of policy-gradient updates on one criterion family. That family's reward is the average of $s$ over its items, excluding items whose conflict $c$ exceeds a threshold $\tau$. Excluded items are sent for re-judging with a paraphrased rubric, and their count is logged.
In the reward block, the policy is held fixed. Fresh rollouts are judged, and three things are recomputed in closed form: each family's $q$, the threshold $\tau$ (a fixed quantile of the conflict distribution), and each family's reliability, defined as one minus its mean conflict.
The next family to train is the one with the highest reliability-weighted headroom. A family the judge currently contradicts itself on is therefore postponed rather than trusted. This replaces ARL-RR's probe-based search over family orderings with a signal read directly off the judge.
The structure borrows from alternating optimization in wireless beamforming, where fixing one block of variables makes the other block easy. Beamforming Design for Beyond Diagonal RIS-Aided Cell-Free Massive MIMO Systems alternates between the transmitter's beam settings and the reflecting surface's settings, and each subproblem becomes tractable once the other block is fixed. We use only that decoupling idea. Our reward block is a cheap closed-form re-estimate, not a convex program, and the policy block is ordinary RL. The inherited caveat also applies: alternation guarantees improvement only toward a stationary point, not a global optimum. We therefore report sensitivity to the initial family order.
Two ideas from this intersection are deliberately set aside.
The first is using $q$ as a continuation dial that smooths the reward landscape early and sharpens it later. Policy Gradient Algorithms Implicitly Optimize by Continuation shows that policy variance acts as a smoothing parameter on the return. $q$ has no comparable property: it changes which judge outputs are admissible and compresses mid-range scores, and it does not convolve the objective with anything. We see no argument that scheduling $q$ performs continuation, so we treat $q$ as a calibration parameter only.
The second is aggregating criteria with a Choquet integral, which weights subsets of criteria to capture redundancy and synergy, as in Higher order hesitant fuzzy choquet integral operator and its application to multiple criteria decision making. That operator needs a measure over up to $2^N$ subsets of criteria. It would also reintroduce the cross-family scalarization that ARL-RR shows is harmful.
The topological reassurance that high-return policies form a connected set, from Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems, holds for the true reward. It says nothing about a reward corrupted by a self-contradicting judge, and that corrupted case is the regime studied here.
Baselines
The naive reference is scalarized RL (SRL): all rubric items averaged into one reward, trained with the same policy-gradient algorithm. ARL-RR reports HealthBench scores for SRL of 0.556 on Qwen3-1.7B and 0.762 on Qwen3-14B. The strongest existing method is ARL-RR itself, reported at 0.576 on Qwen3-1.7B and 0.788 on Qwen3-14B (Alternating Reinforcement Learning with Contextual Rubric Rewards). It is the right competitor because it already solves the scalarization problem and differs from our method only in how it treats judge outputs and orders the families.
All methods are reproduced in our own pipeline rather than compared against the published numbers. They share the same 4,000/1,000 HealthBench split, the same Qwen3-32B judge, the same number of judge calls per step, and the same RL steps and tuning budget. Our method's re-judging is charged against that call budget. Beating ARL-RR means a gain on the 1.7B model, averaged over at least three seeds, that exceeds seed variation. It also means no loss on any of the five families.
The key ablation is a strong confidence-weighted variant of ARL-RR, run under the same budget. Each item gets a single score plus a single uncertainty, taken either from the judge's yes-probability or from the spread of three judge samples, and uncertain items are masked at a matched rate. This isolates the one new ingredient: separating conflict from ignorance.
Further ablations remove individual pieces. One masks on hesitancy (ignorance) instead of conflict. One fixes $q=1$, clipping inadmissible pairs, versus fixing $q=2$ versus fitting $q$. One replaces reliability-driven family ordering with ARL-RR's probe search.
Why not just…?
Why not just weight each item by the judge's confidence? This is the real threat, and it is the main ablation. A scalar confidence treats "no evidence" and "strong evidence both ways" identically. Our hypothesis is that only the second produces exploitable reward. If confidence weighting matches our method at every injected-noise level, the claim is refuted, and we would report that the two-sided representation adds nothing beyond calibration.
Why not just use a better judge, or a majority vote over judge samples? Voting lowers variance but can hide systematic contradiction. A judge that consistently gives an item both high support and high violation will vote the same way every time. We include a five-sample majority-vote ARL-RR at matched cost. The test of whether we are measuring something voting misses is the conflict-injection experiment below, where voting cannot remove the contradiction.
Why not learn scalar weights over criteria, as MORSE does? Automatic Reward Shaping from Multi-Objective Human Heuristics tunes linear weights over heuristic rewards with hypergradients. It needs random restarts because the landscape relating weights to performance is rugged. It also scalarizes, which ARL-RR shows lowers the reward's variance. We do not run it on HealthBench, since it assumes a sparse task-success signal HealthBench lacks, and we treat this as a limitation of the comparison.
Proposed Experiments
The first study is judge-only and costs no RL. On rollouts from an early checkpoint, we check whether conflict and ignorance predict different outcomes. For each item we compare the verdict with one from a paraphrased rubric and with one from a different-size Qwen3 judge. We expect conflicted items to show much lower cross-judge agreement than ignorant items at equal hesitancy. We also measure how often independent elicitation yields pairs outside the $q=1$ and $q=2$ regions. If almost none do, a Pythagorean representation would suffice and $q$ is moot.
The main study trains on HealthBench at 1.7B, with 4B and, compute permitting, 14B. We report the overall score, the five per-family scores, response length, and a hacking indicator: the rate at which a held-out judge flags responses that satisfy and violate the same criterion.
The decisive check is conflict injection. We corrupt a controlled fraction of one family's rubric items with contradictory paraphrases, a setting in which ARL-RR's authors predict error amplification. We then plot the drop in final score against the injection rate for SRL, ARL-RR, confidence-weighted ARL-RR, and our method. The claim survives only if our curve is clearly flatter than the confidence-weighted curve, and the hacking indicator explains the difference.
References
- Alternating Reinforcement Learning with Contextual Rubric Rewards
- Pythagorean fuzzy time series model based on Pythagorean fuzzy c-means and improved Markov weighted in the prediction of the new COVID-19 cases
- Generalized q-ROF TODIM Decision-Making Method Considering Attribute Correlation
- Beamforming Design for Beyond Diagonal RIS-Aided Cell-Free Massive MIMO Systems
- Policy Gradient Algorithms Implicitly Optimize by Continuation
- Higher order hesitant fuzzy choquet integral operator and its application to multiple criteria decision making
- Connected Superlevel Set in (Deep) Reinforcement Learning and its Application to Minimax Theorems
- Automatic Reward Shaping from Multi-Objective Human Heuristics