TL;DR
Instead of querying expensive and opaque LLMs at evaluation time, why not ask them to write the evaluation code instead? This paper introduces PAJAMA, a framework that distills LLM judging logic into a committee of fast, transparent Python programs. PAJAMA matches the accuracy of a 13B LLM judge while delivering orders-of-magnitude higher throughput and slashing API costs by up to 50x.
What Problem Does This Paper Address?
The "LLM-as-a-judge" paradigm has become the standard for automated evaluation, preference labeling, and reward modeling. However, running large models to evaluate millions of samples introduces massive bottlenecks:
- Inference Cost & Latency: High API bills for proprietary models or heavy GPU demands for open-weight deployments.
- Opaque Logic: LLM justifications can be hallucinated, making it hard to verify if a verdict is truly fair.
- Systemic Biases: LLMs notoriously favor longer answers (verbosity bias) or pretty formatting (rich-content bias).
- Re-inference Tax: Tweak a single evaluation criterion, and you have to re-run your entire dataset through the LLM.
The authors tackle these challenges by shifting from model-based inference to synthesized program execution.
Methods
PAJAMA operates via a multi-step pipeline that turns abstract evaluation rubrics into executable code:
- Program Distillation: The system prompts a powerful LLM (e.g., Claude 4.6) once at synthesis time to generate Python functions based on 10 curated rubrics (such as logical coherence, clarity, and completeness). These programs use lightweight, standard Python libraries to extract interpretable features like TF-IDF cosine similarity or discourse markers.
- Output Calibration & Selection: Program scores are normalized and converted into discrete votes ($+1$, $-1$, or $0$ for abstain) using thresholds tuned on a small validation set. Programs performing below random chance are discarded.
- Weak Supervision Aggregation: To resolve conflicts among the remaining program committee, PAJAMA applies a generative label model (inspired by frameworks like Snorkel) to learn per-program weights based on their agreement and disagreement patterns.
- Confidence-Aware Routing: For highly ambiguous cases where the program committee abstains or yields low confidence, PAJAMA routes the sample to an LLM fallback judge.
Datasets
The framework was validated across five distinct pairwise preference datasets:
- JudgeLM
- PandaLM
- MultiPref
- Prometheus
- Preference-700K
For out-of-domain generalization testing, the authors evaluated distilled reward models on RewardBench (covering Chat, Chat Hard, and Reasoning categories).
Key Insights and Ideas
- Code as an Interpretable Medium: Code forces an LLM’s implicit evaluation heuristics into explicit, deterministic algorithms. As seen in Figure 1, an LLM can translate "logical coherence" into concrete checks like sentence-to-sentence semantic drift and explicit penalties for contradictions.
- Bias is a Patchable Bug: Traditional LLM judges suffer from deep-seated biases that are hard to prompt away. In PAJAMA, bias becomes a literal bug in the code. The authors demonstrate that a coding agent (like Claude Code) can inspect the generated Python code, identify superficial biases (like over-rewarding bullet points), and edit the code to fix them.
- Negligible Validation Overhead: While weak supervision often requires a large validation set to estimate accuracies, PAJAMA’s performance remains virtually flat even when shrinking the validation set down to just 5% (a few dozen samples), proving it is highly data-efficient.
Key Results
- Matching 13B Models at 47x Speed: On average across the five datasets, standalone programmatic judges achieved an accuracy of 78.11%—matching the performance of mid-sized open-weight models like OLMo-2-13B-Instruct and Qwen2.5-3B-Instruct—while running at over 439 samples per second.
- Advancing the Pareto Frontier: By using program confidence as a routing signal, PAJAMA pushes the accuracy-throughput Pareto frontier significantly further than standard LLM routing. For instance, paired with OLMo-2-7B-Instruct, it yields a +5.0% accuracy boost at 2.9x higher throughput compared to pure LLM evaluation.
- Superior Reward Signals at 50x Lower Cost: When distilling a reward model on 20,000 samples, using PAJAMA's programmatic labels resulted in a model that outperformed one trained on GPT-4 labels on RewardBench, while slashing the API cost from \$363.97 to just \$7.21.
Limitations and Future Work
- Dependency on Synthesis Capabilities: PAJAMA relies heavily on the underlying LLM's ability to write high-quality, discriminative code during the initial synthesis phase. If the synthesis model struggles to comprehend the evaluation rubric, the resulting code quality degrades.
- Complex Reasoning Constraints: Programmatic judges excel at linguistic, stylistic, and rubric-based evaluations, but they struggle with complex reasoning tasks like advanced mathematics or code verification. For these domains, the authors note that the system must rely more heavily on the hybrid LLM fallback mechanism.
Summary updated