Codifying the Judge: Scalable Evaluation via Program Distillation

Tzu-Heng Huang, Shengqi Qiu, Frederic SalaarXiv 2026 · arXiv

TL;DR

Instead of querying expensive and opaque LLMs at evaluation time, why not ask them to write the evaluation code instead? This paper introduces PAJAMA, a framework that distills LLM judging logic into a committee of fast, transparent Python programs. PAJAMA matches the accuracy of a 13B LLM judge while delivering orders-of-magnitude higher throughput and slashing API costs by up to 50x.

What Problem Does This Paper Address?

The "LLM-as-a-judge" paradigm has become the standard for automated evaluation, preference labeling, and reward modeling. However, running large models to evaluate millions of samples introduces massive bottlenecks:

  1. Inference Cost & Latency: High API bills for proprietary models or heavy GPU demands for open-weight deployments.
  2. Opaque Logic: LLM justifications can be hallucinated, making it hard to verify if a verdict is truly fair.
  3. Systemic Biases: LLMs notoriously favor longer answers (verbosity bias) or pretty formatting (rich-content bias).
  4. Re-inference Tax: Tweak a single evaluation criterion, and you have to re-run your entire dataset through the LLM.

The authors tackle these challenges by shifting from model-based inference to synthesized program execution.

Methods

PAJAMA operates via a multi-step pipeline that turns abstract evaluation rubrics into executable code:

  1. Program Distillation: The system prompts a powerful LLM (e.g., Claude 4.6) once at synthesis time to generate Python functions based on 10 curated rubrics (such as logical coherence, clarity, and completeness). These programs use lightweight, standard Python libraries to extract interpretable features like TF-IDF cosine similarity or discourse markers.
  2. Output Calibration & Selection: Program scores are normalized and converted into discrete votes ($+1$, $-1$, or $0$ for abstain) using thresholds tuned on a small validation set. Programs performing below random chance are discarded.
  3. Weak Supervision Aggregation: To resolve conflicts among the remaining program committee, PAJAMA applies a generative label model (inspired by frameworks like Snorkel) to learn per-program weights based on their agreement and disagreement patterns.
  4. Confidence-Aware Routing: For highly ambiguous cases where the program committee abstains or yields low confidence, PAJAMA routes the sample to an LLM fallback judge.
Figure 2: PAJAMA Workflow. Given a query q and two candidate responses r(1),r(2), a diverse pool of programmatic judges—synthesized by an LLM from curated rubrics—produces initial evaluations. These program outputs are calibrated and selected, and their verdicts are aggregated into a combined decision. Uncertain cases are then routed to an LLM judge to produce the final preference.

Datasets

The framework was validated across five distinct pairwise preference datasets:

  • JudgeLM
  • PandaLM
  • MultiPref
  • Prometheus
  • Preference-700K

For out-of-domain generalization testing, the authors evaluated distilled reward models on RewardBench (covering Chat, Chat Hard, and Reasoning categories).

Key Insights and Ideas

  • Code as an Interpretable Medium: Code forces an LLM’s implicit evaluation heuristics into explicit, deterministic algorithms. As seen in Figure 1, an LLM can translate "logical coherence" into concrete checks like sentence-to-sentence semantic drift and explicit penalties for contradictions.
  • Bias is a Patchable Bug: Traditional LLM judges suffer from deep-seated biases that are hard to prompt away. In PAJAMA, bias becomes a literal bug in the code. The authors demonstrate that a coding agent (like Claude Code) can inspect the generated Python code, identify superficial biases (like over-rewarding bullet points), and edit the code to fix them.
  • Negligible Validation Overhead: While weak supervision often requires a large validation set to estimate accuracies, PAJAMA’s performance remains virtually flat even when shrinking the validation set down to just 5% (a few dozen samples), proving it is highly data-efficient.
Figure 1: An example of synthesizing programmatic judges. Given a synthesis prompt (top-left), an LLM generates a Python judging_function that articulates an evaluation rubric—for example, logical coherence—using interpretable features such as semantic drift via TF-vector cosine similarity or topic consistency across discourse segments. These features are combined into a weighted score, with explicit penalties for contradiction and truncation, where the weights are determined by the LLM itself. This yields a transparent, program-based evaluator distilled directly from an LLM’s reasoning.

Key Results

  • Matching 13B Models at 47x Speed: On average across the five datasets, standalone programmatic judges achieved an accuracy of 78.11%—matching the performance of mid-sized open-weight models like OLMo-2-13B-Instruct and Qwen2.5-3B-Instruct—while running at over 439 samples per second.
  • Advancing the Pareto Frontier: By using program confidence as a routing signal, PAJAMA pushes the accuracy-throughput Pareto frontier significantly further than standard LLM routing. For instance, paired with OLMo-2-7B-Instruct, it yields a +5.0% accuracy boost at 2.9x higher throughput compared to pure LLM evaluation.
  • Superior Reward Signals at 50x Lower Cost: When distilling a reward model on 20,000 samples, using PAJAMA's programmatic labels resulted in a model that outperformed one trained on GPT-4 labels on RewardBench, while slashing the API cost from \$363.97 to just \$7.21.
Figure 3: Effectiveness of Programmatic Judges. Accuracy vs. throughput across five preference datasets. Each subplot title reports the number of programs retained after selection. Programmatic judges achieve extremely high throughput while remaining competitive in accuracy with mid-sized LLM judges—matching OLMO-2-13B-INSTRUCT on average.

Limitations and Future Work

  • Dependency on Synthesis Capabilities: PAJAMA relies heavily on the underlying LLM's ability to write high-quality, discriminative code during the initial synthesis phase. If the synthesis model struggles to comprehend the evaluation rubric, the resulting code quality degrades.
  • Complex Reasoning Constraints: Programmatic judges excel at linguistic, stylistic, and rubric-based evaluations, but they struggle with complex reasoning tasks like advanced mathematics or code verification. For these domains, the authors note that the system must rely more heavily on the hybrid LLM fallback mechanism.

Summary updated

Abstract

LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.
BibTeX
@misc{huang2026codifying,
  author = {Tzu-Heng Huang and Shengqi Qiu and Frederic Sala},
  title = {{Codifying the Judge: Scalable Evaluation via Program Distillation}},
  year = {2026},
  eprint = {2607.22561},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.22561},
}
Figure