PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems

Sven Kleinkop, Tamás Janusko, Ricardo Bochnia, Gunnar Hempel, Steffen Tomschke, Jürgen Anke, Maik ThielearXiv 2026 · arXiv

TL;DR

LLM-as-a-Judge (LLMaaJ) systems often suffer from "judge drift," where their alignment with human evaluation policies degrades as real-world data distributions shift. PromptOps is a new closed-loop framework that treats prompt optimization as a continuous maintenance task rather than a one-shot setup, using iterative recalibration to keep automated judges accurate over time.

What Problem Does This Paper Address?

Large Language Models are increasingly used as automated "judges" to score the outputs of other AI systems (like chatbots) based on criteria like toxicity, bias, or helpfulness. However, these judges are not "set and forget" components. As user behavior changes or application policies evolve, the judge's performance can slip—a phenomenon the authors call judge drift.

Most existing research treats Automatic Prompt Optimization (APO) as a one-shot procedure performed on a static development set. In a live deployment, this is insufficient. Engineers need to know when to recalibrate, which feedback data to use, and how to balance the cost of optimization (API tokens and compute) against the gain in accuracy.

Methods

The authors introduce PromptOps, a modular framework designed to manage the lifecycle of an LLM judge. As shown in Figure 1, it extends the standard LLMaaJ setup with a continuous feedback loop.

Fig. 1: Overview of PromptOps as a closed-loop extension of an LLM judge. At time t, the judge evaluates input item xt using prompt pt and produces evaluation ˆyt. When reference feedback ft is available, the resulting error et and all evaluation artifacts are accumulated in history Ht. The Controller monitors this history, while the Dispatcher, Sample Selector, and Optimizer determine whether and how a new prompt pt+1 is produced.

The framework separates the maintenance process into four distinct operational decisions:

  1. Controller: Determines when to trigger an optimization pass. The authors compared simple periodic triggers against performance-based triggers that monitor error rates in a sliding window.
  2. Dispatcher: Selects the optimization procedure to execute. This can be a single algorithm or a composite pipeline (e.g., running one optimizer after another).
  3. Sample Selector: Decides which historical feedback records (human labels) should be used to train and validate the new prompt.
  4. Optimizer: The engine that actually proposes prompt updates. The study evaluated five methods:
  5. GEPA: An evolutionary method using LLM reflection.
  6. SIMBA: An iterative method that derives rules and demonstrations from execution traces.
  7. Rubric Synthesis: A lightweight custom method that only revises the scoring rubric.
  8. Exemplar Selection (SA/TPE): Methods that optimize the few-shot examples included in the prompt using Simulated Annealing or Tree-structured Parzen Estimators.

Datasets

The framework was evaluated using FairBotBench, which contains 600 German-English e-commerce conversations annotated by humans for bias, toxicity, and personalization.

To test the system's resilience, the researchers induced artificial "judge drift" by creating data streams where the distribution of human scores changed abruptly (e.g., moving from a stream of mostly "good" responses to a stream of "problematic" ones). They tested two judge backends: a proprietary model (GPT-4 family) and an open-weight model (Gemma 4).

Key Insights and Ideas

  • Optimization as a Control Problem: The paper shifts the perspective of prompt engineering from a design task to an operational control problem. The goal isn't just a "perfect" prompt, but a system that can maintain alignment under changing conditions.
  • No Universal "Best" Optimizer: Effectiveness is highly context-dependent. An optimization pipeline that worked wonders for a GPT-based judge often failed to be the best choice for a Gemma-based judge.
  • The Specialization Trade-off: There is a tension between adapting to recent feedback and remaining robust. The authors found that "Rubric Synthesis" was very effective at adapting to specific distributions but was more vulnerable to sudden shifts, suggesting it might be "specializing" too much on past data.
  • Triggering is a First-Class Citizen: When you optimize is just as important as how. Performance-based triggering (responding to a spike in error) proved more effective at limiting the impact of drift than simply optimizing at fixed intervals.

Key Results

  • Recovery from Drift: Iterative optimization successfully recovered judge performance after induced distribution shifts. In some cases, it achieved a 34% to 51% improvement in Mean Absolute Error (MAE) over a static baseline prompt.
  • Pareto Efficiency: The authors mapped out the trade-offs between accuracy (MAE) and cost (cumulative tokens). They found that composite pipelines (like GEPA followed by TPE) often provided the best accuracy for high-resource models, while Rubric Synthesis was a highly cost-effective "Pareto-optimal" choice for both backends.
  • Responsiveness: Performance-based controllers were able to initiate recalibration shortly after a drift occurred, significantly reducing the "error tail" compared to periodic schedules.

Limitations and Future Work

The study has a few notable constraints:

  • Oracle Feedback: The experiments assumed that perfect human feedback was available immediately for every turn. In reality, human labels are often delayed, sparse, and noisy.
  • Scale: The evaluation was limited to 500-turn streams and a single benchmark.
  • Production Readiness: While the framework is a strong abstraction, the authors note it doesn't yet account for the complexities of real-world label acquisition costs or disputed feedback.

Future work will likely focus on how PromptOps handles "imperfect" feedback and how it scales to much longer, multi-domain evaluation streams.

Summary updated

Abstract

Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change. Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time. We frame this problem as judge drift and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems. PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt. We instantiate the framework with five prompt optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection using either simulated annealing or a Tree-structured Parzen Estimator. Our experiments demonstrate that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution. However, effective maintenance is inherently contextdependent: neither optimization algorithm choice nor update timing generalizes universally across settings. These findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps provides a practical abstraction for studying and implementing this process.
BibTeX
@misc{kleinkop2026promptops,
  author = {Sven Kleinkop and Tamás Janusko and Ricardo Bochnia and Gunnar Hempel and Steffen Tomschke and Jürgen Anke and Maik Thiele},
  title = {{PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems}},
  year = {2026},
  eprint = {2611.07123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07123},
}
Figure