Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling

Eugene Agichtein, Charles L. A. Clarke, Kaustubh Dhole, William SternarXiv 2026 · arXiv

TL;DR

Instead of calling a Large Language Model (LLM) to judge every single document in a retrieval pool, ExecJudges uses an LLM once per topic to synthesize a human-readable Python program that performs the grading. This approach matches the accuracy of frontier LLM judges at a fraction of the cost ($0.17–$0.49 per topic) while providing transparent, deterministic, and inspectable code.

What Problem Does This Paper Address?

In Information Retrieval (IR), we evaluate systems by checking how relevant their results are to a specific query. Traditionally, this required expensive human assessors (like those at NIST). Recently, "LLM Judges" have become the standard alternative, but they suffer from three major drawbacks:

  1. Cost and Scalability: You pay for an LLM call for every document. As document pools grow into the thousands, costs explode.
  2. Opacity: LLM judgments are "black boxes." Even with an explanation, it’s hard to verify exactly why a model gave a specific grade.
  3. Reproducibility: LLMs are non-deterministic and can be swayed by "adversarial" content within the documents they are judging.

The authors ask: Can we use the LLM's reasoning to write a standalone program that replaces the LLM itself?

Methods

The authors introduce ExecJudges, a framework that treats the LLM as a "Program Writer" rather than a "Judge."

The Workflow

As shown in the comparison below, the standard LLM judge (a) is a bottleneck, whereas ExecJudges (b) creates a reusable asset.

Figure 1: Relevance assessment with an LLM judge (a) and with ExecJudges (b). The LLM judge is called for every document. ExecJudges calls the LLM once per topic to write programs, and the programs grade all documents.
  1. Program Synthesis: A frontier LLM (Claude 3.5 Opus) is given the topic description, official grading guidelines, and a "tool-use policy." It then writes a Python program to assess documents for that specific topic.
  2. Deterministic Tools: The generated programs are restricted to a "sandbox" and a specific toolkit. They use lexical matching, regex, BM25 scores, and NLP parsers (e.g., for dates or entities) rather than calling neural models.
  3. Consensus Grading: To reduce errors, the system generates three independent programs ($k=3$) for each topic. The final relevance label is the floor of the mean grade across these programs.
  4. ExecJudges-L (Optional): If a few human-labeled examples (e.g., 16 per topic) are available, they are included in the prompt to help the LLM "calibrate" where the lines between relevance grades should be drawn.

Datasets

The method was evaluated across seven diverse TREC (Text REtrieval Conference) collections:

  • TREC Deep Learning (DL) 2019–2023: Modern web search tasks.
  • TREC-COVID: Search over medical abstracts.
  • Robust04: A classic benchmark known for difficult, narrative-heavy topics.

The authors compared ExecJudges against NIST human assessors, pointwise/listwise LLM judges (Claude 3.5 Sonnet/Haiku), and per-topic classifiers.

Key Insights and Ideas

  • Symbolic Distillation: The core "aha" moment is that much of an LLM's relevance reasoning can be distilled into explicit symbolic logic (e.g., "The document must mention 'Salvation Army' AND a founding year like 1865").
  • The "Break-Even" Point: Because writing the program is a fixed one-time cost, ExecJudges becomes cheaper than a listwise LLM judge once a topic has more than ~200–350 documents. For deep pools, the savings are massive.
  • Transparency as a Feature: Because the judge is just Python code, researchers can inspect it to see exactly what criteria are being used. If a program is "missing" a key synonym, a human can simply edit the code.
  • Calibration via Few-Shot Labels: Adding just 16 human labels significantly boosts agreement with NIST, as it helps the LLM understand the specific "strictness" of the human assessors for that collection.

Key Results

  • Accuracy: On TREC DL 2019–2021, ExecJudges achieved agreement with NIST assessors (measured by Krippendorff’s $\alpha$) comparable to the frontier Claude 3.5 Sonnet judge.
  • System Ranking: The labels generated by ExecJudges ranked retrieval systems with a Kendall’s $\tau$ of 0.87 to 0.94 compared to official NIST rankings, which is considered highly reliable for system evaluation.
  • Cost Efficiency: Writing the programs costs between $0.17 and $0.49 per topic. Once written, grading 10,000 documents costs essentially zero in API fees.
  • Superiority over Classifiers: ExecJudges-L outperformed specialized per-topic classifiers (like monoT5-base adapters) when provided with the same small set of 16 labels.

An example of the generated logic for a "Salvation Army" query is shown below:

Figure 2: ExecJudges (Figure 1b) on DL 2019 topic 962179, with the grade formula and the rounding of the mean. Passage 𝑑2 mentions the Salvation Army without its founding date. Rounding down gives grade 0, the NIST grade.

Limitations and Future Work

  • Model Dependency: The quality of the programs depends heavily on the "writer" model. When the authors swapped Claude 3.5 Opus for an open-weight model (GPT-OSS-120B), performance dropped significantly.
  • Complex Narratives: On the Robust04 dataset, which features long documents and complex "narrative" requirements, ExecJudges struggled more than on short-query web search tasks. Simple lexical rules sometimes fail to capture nuanced topicality.
  • Evaluation Gaming: If the grading programs are made public, system builders might "game" the evaluation by specifically optimizing for the regex patterns in the code.
  • Future Directions: The authors suggest exploring active learning (where the LLM asks for specific human labels to clarify its program) and extending the method to long-form generative IR (RAG) evaluation.

Summary updated

Abstract

Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments. To address these issues, we propose ExecJudges, a method for generating fast, inspectable and scalable judges for automatic relevance labeling. Specifically, ExecJudges uses a frontier LLM to turn assessment guidelines into executable programs, which rely only on lexical matching, standard NLP libraries like parsers, and pool statistics. An ExecJudges-L variant also uses examples of human labels if available. We evaluate ExecJudges on seven TREC collections by comparing the produced labels with trained NIST assessors, frontier LLM judges, and recently proposed per-topic classifiers. Our results show that ExecJudges agrees with NIST assessors at the level of the Claude Sonnet 5.5 LLM judges on TREC DL 2019-2021. With a small number of human labels per topic, the ExecJudges-L variant agrees with NIST more than per-topic classifiers with the same labels on all seven collections, and either exceeds the agreement of LLM judges provided the same labels or is within about 0.05 of it in Krippendorff's alpha. The labels rank the official TREC DL runs with Kendall's tau of 0.87 to 0.94, and writing the programs incurs a one-time cost of $0.17 to $0.49 per topic, with no additional costs for grading. Our results suggest that ExecJudges offers a reproducible, inspectable, efficient and cost-effective alternative to LLM judges for labeling deep result pools for system evaluation or ranking improvements.
BibTeX
@misc{agichtein2026can,
  author = {Eugene Agichtein and Charles L. A. Clarke and Kaustubh Dhole and William Stern},
  title = {{Can Programs Replace LLM Judges? Executable Program Judges for Scalable and Efficient Relevance Labeling}},
  year = {2026},
  eprint = {2611.24849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24849},
}
Figure