When teams build an automated judge (a large language model prompted to grade other models' outputs), they usually write a grading rubric by hand while looking at three to five examples. The rubric fits those examples and then fails on the full dataset. Practitioners using IBM's EvalAssist tool call this the "over-fitting trap" (EvalAssist). Programming by example, the field that infers programs from a handful of input-output pairs, has studied this failure for decades under the name under-specification: a few examples are consistent with many programs, and the learner tends to pick a spurious one. That field has two standard remedies. The first is to search simple programs before complex ones. The second is to ask the user about inputs on which the surviving candidate programs disagree. Neither remedy has been applied to building evaluators.
This proposal treats a rubric as a small program. Its primitives are yes/no checks, each answered by an LLM that must quote the evidence behind its answer, and the checks are combined by simple logic. We learn such a program from human verdicts using three components. Search moves from simple rubrics to complex ones. The system asks for new labels only on the outputs where the rubrics still consistent with the data disagree. A cheap automatic verifier rejects any check whose answer flips under changes that should not matter, such as adding a fake citation, a "NEW (2025)" tag, or padding. The claim is that this recovers the agreement with human judgment of hand-locked rubrics using a fraction of the labels, and that it is measurably harder to fool with known judge shortcuts.
Core Claim
Much of what goes wrong in current evaluator induction is an ambiguity problem, not a capacity problem. AutoMetrics generates many candidate LLM evaluators and fits a linear weighting of them to human labels. It needs about 80–100 labels before performance saturates, and it chooses which items to label at random. Its components are free-form prompts, so shortcut biases inside them can pass straight into the fitted weights. RULERS shows that turning a rubric into a locked specification with evidence requirements raises agreement sharply: quadratic weighted kappa on ASAP 2.0 essays goes from 0.4319 to 0.7276. But a human expert must write that rubric first.
We claim three things. First, a rubric grammar that makes search order meaningful, meaning evidence-gated binary checks combined by bounded logic, lets us borrow the idea from Overfitting in Synthesis that simple grammars should be exhausted first. That paper proves more expressive grammars overfit more; on invariant synthesis, a richer grammar raised counterexample-guided refinement rounds on one problem from 19 to 88. Second, choosing which output to label by how much the consistent rubrics disagree about it will reach a given human correlation with substantially fewer labels than random labeling. Third, known judge biases give us something ordinary evaluation lacks: a test oracle.
The third point needs explaining. Is Programming by Example Solved by LLMs? relies on running candidate programs against input-output pairs to check them, and that paper's authors note no such oracle exists for text quality. There are, however, invariances, changes that should never alter a verdict. The bias literature catalogs them. Justice or Prejudice? lists twelve, including fabricated citations, verbosity, position and sycophantic tone. The Silent Judge adds recency and provenance tags, showing GPT-4o shifts 30% of verdicts on ELI5 under recency cues while acknowledging the cue 0% of the time. Each invariance gives an automatic counterexample generator for the synthesizer.
Research Program
Rubric grammar. A rubric is a set of checkpoints combined by a composition rule. A checkpoint is a natural-language yes/no question answered by a fixed, modest judge model. Following EvalAssist, the answer is read from token log-probabilities rather than parsed text. Following RULERS, the judge must return a verbatim supporting quote, and a "yes" without a quote that can be matched in the text is treated as "no."
The grammar has three levels:
- Level 1: a single checkpoint.
- Level 2: conjunctions or disjunctions of up to three checkpoints. This is the disjunctive-normal-form rule format that EmFORE learns from email-filing demonstrations.
- Level 3: a sparse weighted threshold over up to eight checkpoints.
Proposing checkpoints. An LLM proposes candidate checkpoints from contrasting pairs of labeled outputs, one rated high and one rated low. This is the same generation step AutoMetrics uses, but restricted to evidence-gated binary questions.
Search cost. Each checkpoint is run once on every labeled item, every unlabeled item and every perturbed item, and the results are cached. The search itself then operates over a binary feature matrix and needs no further model calls. This avoids the minutes-per-query rejection-sampling cost reported in the LLM programming-by-example work.
Version space and query selection. Let $L$ be the set of labeled pairs $(x, y)$, where $x$ is a model output and $y$ is the human verdict. Let $\mathcal{R}_k$ be the rubrics expressible at grammar level $k$. The version space is
$$V_k(L) = \{\, r \in \mathcal{R}_k : \mathrm{err}_L(r) \le \epsilon \ \text{and}\ r \text{ passes all invariance tests on } L \,\},$$
where $\mathrm{err}_L$ is the error on $L$ and $\epsilon$ is a noise tolerance set from inter-annotator disagreement. Search stays at the lowest level $k$ for which $V_k(L)$ is non-empty. This is hybrid enumeration applied to rubrics. The next item sent to the human is the unlabeled output that maximizes the entropy of the verdict vote among the members of $V_k(L)$, which is the programming-by-example "distinguishing input" query.
Deployment and abstention. At test time the system returns the verdict of the simplest surviving rubric. When surviving rubrics disagree beyond a threshold, it abstains. This mirrors EmFORE's suppression mechanism and yields a coverage–accuracy curve rather than a single number.
Invariance verifier. For every labeled item we generate perturbed versions using the CALM attacks (fake authority citations, length padding, polite or sycophantic rewording, position swap in pairwise tasks) and the Silent Judge cues (recency and provenance tags). A checkpoint whose answer flips on more than a small fraction of these pairs is removed before search. Flipped items are also added as counterexamples labeled with the original verdict, which closes the counterexample-guided loop without human effort.
Soundness of the perturbations. We restrict ourselves to perturbations the source papers designed to be non-substantive. We will also hand-audit a sample, because some perturbations, such as padding, can occasionally change content.
Baselines
Direct judge, no human labels. The naive reference is the same judge model prompted with the task description and a generic quality rubric. On LLMBar's adversarial set, the paper's authors report about 73% accuracy for vanilla GPT-4 (Evaluating LLMs at Evaluating Instruction Following). We rerun this reference with our judge model rather than rely on the reported figure.
Few-shot judge. The same judge model gets every label the budget allows as in-context examples. This is the practitioner default EvalAssist describes.
AutoMetrics. This is the strongest existing competitor, because it solves exactly our problem: building an evaluator from under 100 human labels. We run it with identical label sets and the same judge model for its generated evaluators. It keeps its MetricBank retrieval, and its candidate-generation budget matches our checkpoint-proposal budget in LLM calls. The file reports a relative gain for AutoMetrics on EvalGen (Kendall correlation improved 33.4% over the closest baseline) but no absolute value. No reported result was found on Lacuna for AutoMetrics' absolute Kendall correlation on EvalGen, SimpEval, RealHumanEval or CoGym, so we reproduce it. Beating it means matching its held-out Kendall correlation at 100 random labels while using at most 40 actively chosen labels, and having a lower verdict-shift rate under perturbation at equal label budget.
Hand-written locked rubric. Where expert rubrics exist, RULERS is the ceiling for "a human wrote the specification." Its reported GPT-4o-mini kappa on ASAP 2.0 is 0.7276. On LLMBar, the best hand-designed prompting strategy (Rules + Metrics + Reference) reaches 82.8% on the adversarial set with GPT-4. Reaching within a few points of these without human rubric authoring would count as success.
Ablations. Four ablations isolate the key ingredients. The first uses random rather than disagreement-driven queries. The second starts search at level 3, removing the simple-first ordering. The third drops the invariance verifier. The most important one adds our two ingredients to AutoMetrics: bootstrap-disagreement querying over its fitted weights, and the same perturbation filter applied to its candidate evaluators. That last ablation separates the value of the discrete, evidence-gated grammar from the value of the querying and verification loop.
Why not just…?
Why not just put the labeled examples in the prompt? EvalAssist documents that rubrics tuned on a few examples collapse on larger corpora. LLMBar shows that unstructured prompting of strong models is fooled by adversarial distractors, with LLaMA-2-70B and ChatGPT near chance. The few-shot baseline is run at every label budget, so if it matches us the learning curves will show it directly.
Why not just use AutoMetrics and label more? This is the real threat. At 100 or more labels AutoMetrics may equal us on correlation. Our claim rests on the low-budget part of the learning curve and on robustness, not on the saturated regime. The AutoMetrics-plus-querying-plus-filter ablation settles whether the discrete grammar contributes anything. If it does not, we will report the contribution as the query and verifier loop alone, which transfers to any evaluator ensemble.
Why not just tell the judge to ignore length, citations and labels? The Silent Judge finds a 0% cue-acknowledgment rate: judges do not report the cues that drive their verdicts, so an instruction to ignore those cues has no evident handle to act on. LLMBar also finds that step-by-step reasoning can make adversarial accuracy worse. We test this directly with a debiasing-instruction variant of the direct judge, measured by verdict-shift rate under the same perturbations.
Why not just fine-tune a reward model? HelpSteer2 needed about 21,000 multi-attribute annotations to reach 92.0 on RewardBench and to control length bias. Our regime is tens of labels for a new task, and the resulting rubric can be audited check by check. Fine-tuning is therefore out of scope, not a competitor at equal budget.
Proposed Experiments
Tasks and labels. We use four sources of human verdicts:
- LLMBar, using its official repository and 419 pairwise instances, reporting natural and adversarial accuracy separately;
- the AutoMetrics tasks EvalGen, SimpEval, RealHumanEval and CoGym, reporting Kendall correlation;
- ASAP 2.0, reporting quadratic weighted kappa;
- HelpSteer2's correctness and helpfulness ratings as a large pool from which labels are revealed on request.
Every method starts from the same five seed labels and draws from the same unlabeled pool, with budgets of 10, 20, 40 and 100 labels. The judge model is shared across methods, and results are averaged over five seeds.
Robustness. On ELI5 and LitBench outputs, we inject the Silent Judge recency and provenance cues and the CALM authority, verbosity and sentiment perturbations. We report the verdict-shift rate for every learned evaluator, applying the perturbations to held-out items not used by the verifier. To test generalization rather than memorization, one bias family is held out entirely from the verifier.
Label efficiency. The decisive check is the label-efficiency curve. The method succeeds if, at 20–40 labels, it matches AutoMetrics at 100 labels on at least three of the five correlation tasks, with a lower verdict-shift rate. It fails if the AutoMetrics-plus-querying-plus-filter ablation closes the gap, which would show that the program structure itself does not help.
Secondary checks. Three further analyses round out the evaluation:
- whether search levels chosen by simple-first ordering generalize better than level-3 rubrics fitted on the same labels, confirming the overfitting theory in this setting;
- whether abstention concentrates errors, measured as accuracy at 80% coverage against full coverage;
- a small user study in which practitioners compare synthesized rubrics against their own EvalAssist-style drafts for held-out agreement.
References
- EvalAssist: LLM-as-a-Judge Simplified
- AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
- RULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation
- Overfitting in Synthesis: Theory and Practice
- Is Programming by Example Solved by LLMs?
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- EmFORE: Learning Email Folder Classification Rules by Demonstration
- Evaluating Large Language Models at Evaluating Instruction Following
- HelpSteer 2: Open-source dataset for training top-performing reward models