Teaching Agents What Words Mean by Watching Experts Judge Their Videos

An expert can usually say how a long task should go ("open the drawer, then pick up each box, and never drop one") and can tell at a glance whether a video of a robot did it right. What the expert cannot easily provide is the thing reinforcement learning actually needs: a reward function over raw camera pixels, or thousands of frames hand-labeled with facts like "is holding the red box." We propose to close that gap with three pieces. A language model drafts a reward machine from the expert's description. A reward machine is a small state machine whose states are task stages and whose edges fire when certain facts become true. The expert then watches short videos of the agent and gives pass/fail verdicts with occasional time-stamped critiques. Those verdicts train a pixel-to-fact classifier through the state machine, so the classifier learns to fire facts in the order that would make passing videos pass and failing videos fail. No one ever labels an individual frame.

The second idea is the hierarchical one. The low-level controllers learn a separate value estimate for each fact: how reliably they can make that fact come true. When a stage of the state machine looks unreachable, either the controller cannot do it or the classifier misreads the scene. The agent spends its limited budget of expert questions on exactly those stages. The claim is that this loop reaches the task success of current pixel-grounded reward-machine agents with a small fraction of the expert effort they require. The loop should also outperform zero-shot vision-language labelers, which need no expert at all.

Where This Sits

The closest work splits the problem in ways that each leave one part manual. ARM-FM uses a foundation model to write reward machines and Python labeling functions from task descriptions. Those functions read internal simulator variables, not pixels, and on Meta-World they needed human repair at edge cases. ARM-FM supplies our drafting step, and we replace its state-reading labelers with learned visual ones. Ground-Compose-Reinforce (GCR) does ground facts from pixels and introduces compositional reward shaping. In that shaping, value functions pretrained for each atomic fact are combined to give dense rewards for a whole state machine. GCR's grounding, however, depends on roughly 350 trajectories with human-annotated truth values for a fixed vocabulary. It is our strongest baseline, and the main cost we aim to remove. Visual Reward Machines (VRM) make the state machine differentiable with a temperature-controlled softmax and show that its temporal structure filters single-frame classifier noise. Its automata come from fixed training sequences, and its symbols are unnamed indices. We reuse its relaxation, but drive it with expert verdicts and named, language-described facts.

On the elicitation side, Roleplay-doh has experts critique interactive rollouts and compiles the critiques into explicit principles. It reports 6.04/7 versus 5.24/7 authenticity for simulated patients. This motivates our interaction design: experts critique behavior instead of writing specifications. But Roleplay-doh is text-only and never touches perception. OntoChat shows that unguided language models give shallow specifications without structured scaffolding. That finding constrains us to put the expert's effort into judging rollouts rather than prompting. KIM turns advice into sparse networks with named latent variables, reaching about 90% on Lunar Lander. It works from state inputs and has no temporal hierarchy. We share its bet that named intermediate quantities make expert knowledge usable.

Finally, DIPPER shows that hierarchical agents fail when the upper level proposes subgoals the lower level cannot reach. It fixes this by regularizing the upper level with the lower level's value function, but only over numeric state-vector subgoals. We carry that feasibility signal into symbolic stages. Instead of penalizing the planner, we use it to decide where to ask the human.

Core Claim

Episode-level expert judgments, routed through a language-model-drafted state machine, carry enough information to train a pixel-level fact classifier. The value functions of the low-level controllers then identify which facts are under-grounded or infeasible, and this cuts the number of expert queries needed.

Research Program

Grounding is learned as follows. Let $g_\theta(o_t)$ be the classifier's probabilities for each named fact at frame $o_t$. Let $P_\theta(\text{accept}\mid o_{1:T})$ be the probability that the relaxed state machine ends in its success state on an episode of $T$ frames, computed with VRM's softmax relaxation. Let $y_e \in \{0,1\}$ be the expert's verdict on episode $e$. We train

$$ \mathcal{L}(\theta) = -\sum_e \big[y_e \log P_\theta(\text{accept}\mid o^e_{1:T}) + (1-y_e)\log(1-P_\theta(\text{accept}\mid o^e_{1:T}))\big] + \lambda\, \mathcal{L}_{\text{crit}}(\theta). $$

Here $\mathcal{L}_{\text{crit}}$ is a cross-entropy on the few facts that a language model extracts from time-stamped critiques. For example, "it let go of the box around here" becomes a fact-false label inside a short time window, and $\lambda$ weights this term.

Sequence-level labels alone suffer from a known identifiability problem: two facts can swap meanings and still explain every verdict. We break the symmetry by initializing $g_\theta$ from a vision-language model's similarity to each fact's language description. The critique term then anchors the facts that remain ambiguous. Whether this is enough is the main empirical risk, and we test it directly.

Control follows GCR. We train one value function per fact from online rollouts rewarded by the current classifier. These are combined into shaped rewards for the full state machine, and low-level policies are trained with proximal policy optimization (PPO). Because these per-fact value functions come from online rollouts rather than dedicated demonstrations, the method also addresses GCR's need for per-fact pretraining data.

For querying, consider each state-machine edge. When the combined value from states where the classifier says the source stage is active stays low, the agent selects rollouts near that edge and shows them to the expert. The expert's verdicts then distinguish two cases: the classifier mislabels the scene, which a relabel or critique fixes, or the controller genuinely cannot do the step. In the second case the language model is asked to revise the state machine, for example by inserting an intermediate stage.

Baselines

The naive reference point is flat PPO from pixels with only the sparse task-success reward. ARM-FM reports that flat RL "failed to make any progress" on comparable compositional tasks, so we expect near-zero success. We include it only to confirm the tasks are hard.

The strongest existing method is GCR, run from the official codebase with its 350 annotated trajectories. GCR reports 1.0 success on DrawerWorld Pickup-Each-Box with 350 pretraining trajectories. It drops to 0% without compositional shaping, and a non-compositional temporal-logic-conditioned behavior cloning baseline scores about 0 (Ground-Compose-Reinforce). Matching 1.0 is not the bar, since GCR already saturates success. Beating GCR means reaching at least 0.95 success at a matched environment-step budget while using at most one tenth of GCR's expert effort. Expert effort is measured as simulated-expert seconds, counting a frame label and an episode verdict at costs calibrated in our pilot human study. Both methods share the classifier backbone, PPO hyperparameters, and tuning budget.

For GeoGrid, no reported result was found on Lacuna, so we will run GCR ourselves. For Visual Minecraft, VRM reports "near-perfect" offline sequence accuracy against 40% for CNN+LSTM; we use VRM trained on fully labeled sequences as an upper reference.

An oracle baseline uses ARM-FM-style labelers that read simulator state, giving the ceiling achievable with perfect grounding.

Ablations isolate each ingredient. Removing $\mathcal{L}_{\text{crit}}$ tests whether verdicts alone suffice. Removing the vision-language initialization tests the symmetry breaking. Replacing feasibility-guided query selection with random rollout selection at equal query count tests the hierarchical contribution. The last ablation is the cleanest test of whether the value-function signal matters.

Why not just…?

Why not just use a vision-language model as a zero-shot labeler for each fact, with no expert at all? This is the most serious threat, because it costs zero expert time. Work on the modality gap in contrastive models (Mind the Gap; Learning Robust Visual-Semantic Embeddings) suggests image-text similarities are poorly calibrated for fine distinctions like "gripper holding box" versus "gripper touching box." VRM's results suggest per-frame noise breaks state-machine progress. Still, we run this exact system: the same pipeline with thresholds tuned on a small validation set. If it matches our method on DrawerWorld, the expert loop is unnecessary for these tasks. We would then report that honestly and shift the claim to whatever task regime the zero-shot labeler fails on.

Why not just learn a reward model from the expert's verdicts directly, as in preference-based RL, and skip the state machine? The verdicts would then train an unstructured video reward model. The CNN+LSTM result in Visual Reward Machines, 40% versus near-perfect, and GCR's near-zero non-compositional baseline both suggest that unstructured sequence models need far more labels for multi-stage tasks. We include a recurrent video reward model trained on the same verdicts and critiques at the same query budget to settle this.

Why not just have experts label frames, as GCR does, but fewer of them? Frame labeling may be cheap per label. We therefore sweep GCR's annotation budget downward, at 350, 100, 35, and 10 trajectories, and compare effort-matched curves. If GCR with 35 trajectories matches us at equal seconds of expert time, episode verdicts offer no advantage, and the paper becomes a negative result about weak supervision.

Proposed Experiments

Main experiments run on DrawerWorld and GeoGrid via the GCR codebase, with Visual Minecraft for offline grounding accuracy. Experts are simulated from simulator state. Verdicts are flipped at a 0–20% noise rate, and a language model paraphrases the true fact changes into critiques. A pilot with roughly ten human participants calibrates the time costs and checks that real critiques resemble the simulated ones.

Metrics are task success, per-fact classifier F1 against simulator truth, expert seconds, and number of state-machine revisions.

The decisive checks are three. First, the expert-effort-versus-success curve must dominate both GCR's reduced-annotation curve and the zero-shot vision-language labeler. Second, the feasibility-guided query ablation must show a measurable effort reduction over random query selection. Third, an induced-infeasibility test removes a stage from the drafted state machine. The query mechanism should localize the missing stage and trigger a repair, which no baseline can do.