Letting a Person Be the Fitness Function: Batch-Optimal Preference Queries for LLM Reward Synthesis

Systems that have a large language model write reward code for robots now beat hand-written rewards on most simulated benchmarks. They all rely on a hidden crutch: a programmatic success checker that scores each candidate reward by training a policy and measuring task success. For the tasks where reward engineering is hardest, such as "drive politely," "move the limb smoothly," or "hand over the cup the way this user likes," no such checker exists. The only fitness signal is a person watching rollouts, and a person can only make a few hundred comparisons before fatigue sets in. We propose to replace the success checker with a small, carefully chosen batch of human preference comparisons. The comparisons are picked by classical optimal experimental design, so that each batch covers different behaviors and each comparison is actually hard to call.

The trick that makes this tractable is to reuse the LLM's own reward programs as the feature space for learning the person's preferences. Every candidate program is a sum of interpretable terms: distance to goal, jerk, joint torque, lane offset. Evaluating every term from every program on a trajectory gives a feature vector that no engineer had to design. A linear preference model over these features can be fit from very few comparisons. Its information matrix then says which comparisons to ask next, which candidate rewards are winning, and which behavioral directions the program library does not yet distinguish. That last signal goes back to the LLM as reflection feedback.

Core Claim

Batch active preference learning has always assumed a hand-built feature map $\phi$ and a linear reward $w^\top\phi(\xi)$ (Batch Active Learning of Reward Functions from Human Preferences, Batch Active Preference-Based Learning of Reward Functions, APReL). This moves the burden of reward engineering onto feature engineering. LLM reward synthesis (Eureka, RF-Agent, LaRes) removes the feature-engineering burden but needs an oracle fitness score.

We claim the two halves fix each other. The program library supplies $\phi$. Diversity-aware batch design over that $\phi$ supplies a fitness signal cheap enough for a human to provide. Concretely, we predict that with a few hundred simulated-human comparisons per task, the loop recovers most of the performance Eureka reaches with oracle fitness. We also predict that it clearly beats the same loop with random pairs, top-k uncertainty pairs, or a generic neural reward model with last-layer design. The contribution is not "add a DPP to Eureka." It is a closed loop in which the human's information matrix decides queries, program fitness, and what the LLM should write next.

Mechanism

At round $t$ the library holds programs $P_1,\dots,P_K$. Each program has been used to train a policy, exactly as in Eureka, and we keep rollouts from all policies. For a trajectory $\xi$, let $\phi(\xi)\in\mathbb{R}^d$ stack the trajectory-summed, standardized output of every term of every program. A comparison between trajectories $\xi_a$ and $\xi_b$ is modeled with the Bradley-Terry choice rule:

$$p(\xi_a \succ \xi_b \mid w) = \sigma\big(w^\top(\phi(\xi_a)-\phi(\xi_b))\big),$$

Here $\sigma$ is the logistic function and $w$ is the unknown preference weight vector. We fit $w$ with a Gaussian prior and choose the next batch $S$ of $B$ pairs by past-aware D-optimality:

$$S^\star=\arg\max_{|S|=B}\ \log\det\Big(H_{\text{past}}+\sum_{(a,b)\in S}\sigma_{ab}(1-\sigma_{ab})\,x_{ab}x_{ab}^\top\Big),$$

In this objective, $x_{ab}=\phi(\xi_a)-\phi(\xi_b)$, $\sigma_{ab}$ is the current predicted preference probability, and $H_{\text{past}}$ is the Fisher information from comparisons already answered. The weight $\sigma(1-\sigma)$ favors pairs that are neither obvious nor hopelessly close. The determinant favors pairs whose differences point in new directions. This is the same coupling of informativeness and diversity that Active Reward Modeling exploits on frozen LLM embeddings. We reuse its greedy, Taylor-expanded solver, including cross-prompt anchoring, which here becomes cross-policy comparisons.

The fitted posterior then plays three roles. First, each program's fitness is the expected preference score of its policy's rollouts, and this replaces the oracle in the evolutionary step. Second, the LLM's reflection prompt receives the learned term weights and the directions of $\phi$ with the least information, i.e. the smallest eigenvectors of the information matrix, expressed through their term names. Third, before spending RL compute on new candidates, we use their predicted rollout features to pick a subset expected to add the most information. These predictions come from cheap short training runs, or from relabeling stored rollouts with each candidate's terms in the style of LaRes buffer relabeling. This applies the same diversity-informativeness tradeoff to policies rather than queries.

One known risk is misspecification: the person may care about something no program term measures. The theory that justifies batch design assumes realizability (Achieving Minimax Rates in Pool-Based Batch Active Learning). We therefore track held-out preference accuracy, and a persistent plateau triggers an explicit "propose a new term" prompt. Active Inverse Reward Design motivates this. It showed that asking designers about trade-offs between candidate rewards reveals what their proxy omitted. Its queries were sequential and its features fixed, whereas ours are batched and the features grow.

Research Program

We will build the loop on top of Eureka's open IsaacGym pipeline and keep its policy-training budget per round unchanged. Any compute differences then come only from query selection, which is cheap. We will implement the selector on APReL's preference-model interface, extended to a feature dimension that grows across rounds.

The analysis has three parts. The first is label efficiency: how many comparisons the method needs to match oracle-fitness Eureka. The second is mechanism: whether the gains come from the program-derived features, from the batch design, or from the feedback to the LLM. The third is honesty about failure: how often misspecification plateaus occur and whether new-term prompting resolves them.

Baselines

The naive reference is human-fitness Eureka with random pairs. It runs the same loop, but each round's comparisons are drawn uniformly among rollouts of current candidates. It gets the same label budget, the same LLM, and the same RL steps. In reward modeling for LLMs, random pairing needed about 20,000 comparisons to match about 1,000 D-optimal ones (Active Reward Modeling). In online driving preferences, random queries gave 7.94% versus 11.3% speed error against active EVOI (Active Reward Learning from Online Preferences). Random pairing is therefore expected to be weak, but it sets the floor.

The upper reference is oracle-fitness Eureka and RF-Agent, run with each task's programmatic success metric. RF-Agent reports an average normalized score of 2.68 on 7 IsaacGym tasks, against 2.00 for Eureka, where 1.0 is the human expert reward (RF-Agent). Eureka reports beating human rewards on 83% of tasks (Eureka). We do not expect to beat the oracle. Success means recovering a stated large fraction of the oracle's normalized score, which we pre-register as at least 80% of Eureka's, under a budget of a few hundred simulated comparisons per task.

The strongest competitor that uses no hand-built features is a neural reward model with last-layer D-optimal selection. This is the Active Reward Modeling method transplanted to robot trajectories: a small network on raw state sequences, last-layer Fisher design, and past-aware batching. It is the most credible way to learn preferences without engineered features. It gets the same pool, the same labels, and a matched hyperparameter search budget. Beating it at equal labels, by a margin larger than seed variance across 5 seeds, would show that the program-derived features carry real value.

Two classical batch selectors run over the same program features: DPP selection (Batch Active Learning of Reward Functions from Human Preferences) and successive elimination (Batch Active Preference-Based Learning of Reward Functions). These isolate whether D-optimality specifically matters beyond diversity in general.

Two ablations target the key ingredients. The first removes the feedback channel: the information matrix still picks queries and scores programs, but the LLM sees only fitness. The second removes the selection of which candidates get trained.

Why not just…?

Why not just let a vision-language model judge rollouts instead of a person? This is the most serious alternative. CoRe already combines vision-language-model preference feedback with rewards, and no reported result found on Lacuna for its gap to oracle fitness on IsaacGym. For tasks with an obvious visual goal, a vision-language judge may be enough. It cannot know a specific user's style, though, such as aggressive versus defensive driving in the Biyik user study. We will run a vision-language-judge variant inside our loop, and as a standalone fitness function, on both goal-type and style-type tasks. If it matches our method on style tasks, the human-in-the-loop claim narrows to goal tasks, and we will report it that way.

Why not just raise the sampling temperature, or deduplicate reward code by code embeddings? Code that looks different can produce identical behavior, for example by rescaling a coefficient or swapping an L1 norm for an L2 norm. Code that looks alike can also diverge after training. Our diversity is measured in the space of behavior the person judges. A baseline that runs a DPP over code embeddings to choose which candidates to train, with random queries, tests this directly. If it matches the behavior-space selector, the policy-selection ingredient is unnecessary, and the ablation will show it.

Why not just ask the person to pick the best video each round, with no preference model? This is top-1 selection with greedy uncertainty in disguise. It extracts at most one bit-like signal per round and repeats near-identical choices, the redundancy documented by Batch Active Preference-Based Learning of Reward Functions and the online driving study. It also gives the LLM nothing about which terms matter. We include it at matched human time, counting a best-of-K choice as K−1 comparisons.

Proposed Experiments

The main benchmark is the 7 IsaacGym tasks used by RF-Agent, with simulated humans. Preferences are drawn from a Boltzmann choice model on each task's true success metric, at several noise levels. We report normalized score against comparisons used, at budgets from 50 to 500.

For harder dexterity, we run the Bi-DexHands Expert-Hard tasks, where RF-Agent reports earlier LLM methods "failed to reach even half of human performance." This tests whether human-fitness search degrades more than oracle search on difficult tasks.

For tasks with no programmatic fitness, we use the Driver environment and highway-env with stylistic targets. These have simulated users first and then a human study of about 20 participants, mirroring Active Reward Learning from Online Preferences. We measure speed and style adherence error and perceived intelligence on the same 7-point scale.

The misspecification probe hides from the LLM's prompt the one term the simulated user cares about. We then measure whether held-out preference accuracy plateaus and whether new-term prompting recovers it.

The decisive check is the comparison against the neural last-layer D-optimal baseline at equal labels on IsaacGym. If program-derived features do not beat it, the core idea fails. That idea is that the LLM's reward terms are a good basis for learning what a person wants, and the remaining contribution would shrink to a better query selector. If they do beat it, and the feedback ablation also costs performance, the loop is doing what we claim.