Calibrated Teacher Calls for On-Policy Agent Training

When a small language-model agent is trained to act in multi-step environments such as web forms, terminals, or text-based household tasks, the most useful supervision comes from a larger teacher model that steps in at the moments where the student has gone wrong. This is hybrid imitation-reinforcement learning: the student learns partly by copying a demonstrator and partly by trial and error against task reward. Every teacher call costs real inference compute or API spend, and current methods are wasteful about when they call. Some switch to the teacher at a random step. Some query it at every state. Some wait until an entire batch of attempts has failed. We propose to decide when to call the teacher, and when to hand control back, using a confidence gate. The gate is a learned estimate of how likely the student's next action is to be wrong. A small random sample of teacher "audit" queries keeps it calibrated, and its threshold is set by the cost of a teacher call.

The point that makes this more than "threshold the entropy" is that the student's confidence signal drifts during training. Reinforcement learning sharpens the policy, so the student's uncertainty (the entropy of its action distribution) shrinks. It also shrinks on states where the student is confidently wrong. A fixed entropy cutoff stops firing as training proceeds. A fixed top-percentile cutoff keeps firing on whatever happens to be least certain, which may be harmless. Semi-supervised learning met the same problem with pseudo-labels and answered it with thresholds that track the shifting distributions of correct and incorrect predictions. We carry that answer into the interactive setting, where the decision is no longer "keep this label" but "spend teacher tokens here."

Core Claim

Our claim is that a cost-aware, audit-calibrated gate will reach a given task success rate with substantially fewer teacher tokens than random switching, dense querying, or a fixed entropy threshold. It should also beat a fixed teacher continuation length. The advantage should grow over multiple rounds of on-policy training, because that is when the relationship between entropy and error drifts. If a fixed entropy percentile matches the calibrated gate across rounds, then drift does not matter in this regime, and we will report that.

Grounding in Prior Work

The closest work is A Few Teacher Steps Go a Long Way (OP-SHORT). It lets the student act on its own, switches to the teacher at a random step, and keeps a fixed-length teacher continuation of $K$ steps without filtering by outcome. It shows that this beats plain behavioral cloning at matched supervision budgets. It also shows that a variant which discards teacher continuations that fail the task (OP-SUCCESS) wastes large amounts of teacher tokens. Its results are non-monotone in $K$: on HotpotQA, $K=3$ beat $K=5$. The authors list the lack of a principled $K$ as a limitation, and the study covers only one round of data collection. This paper supplies our cost accounting, which separates teacher inference cost from student training cost, and our benchmarks. We change two things: when the switch happens, and when it ends. We also run multiple rounds.

Three other papers trigger supervision without calibration. AdaNav shows that action entropy picks out moments of confusion in navigation. It invokes reasoning on roughly 2.5 of 55 steps and concentrates rollouts on hard trajectories. However, it triggers the agent's own internal reasoning, not an external teacher, and it compares against fixed entropy criteria. This motivates our signal choice, and its entropy rule is our main heuristic competitor. TAPO injects teacher answers only when a whole group of reinforcement-learning rollouts gets identical rewards, which is a coarse, after-the-fact trigger. FA-OPD queries its teacher at every visited state. Together these mark the two extremes our gate sits between.

The thresholding method comes from DAW. It frames pseudo-label selection as minimizing the sum of two errors: keeping wrong labels and discarding right ones. It proves that the optimal weighting is a hard step at the point where the confidence densities of correct and incorrect predictions cross, with those densities tracked by an exponential moving average. DAW also constrains our design. Its proof assumes truncated Gaussian densities, which the file flags as fragile for multi-modal confidence distributions. We therefore fit the error probability nonparametrically. A likelihood-ratio threshold stays optimal for a two-error cost without any Gaussian assumption.

The training schedule comes from How to Train Your LLM Web Agent. It shows that branching early from supervised fine-tuning into reinforcement learning with GRPO (group relative policy optimization) is compute-efficient. It also reports that list-filtering tasks in WorkArena stay at 0% success. We use that failure as a stress test: do targeted teacher calls reach states that uniform data scaling never covers?

Research Program

The student is Llama 3.1 8B and the teacher is Llama 3.3 70B, the pairing used by Vattikonda et al. Training alternates rounds. In each round the student rolls out, the gate decides where the teacher intervenes, and the resulting mixed trajectories go into a supervised update followed by GRPO on task reward.

At each student state, we compute cheap student-side features $s$: action entropy, the gap between the two most likely actions, step index, and a short moving average of entropy. The moving average is there to separate sustained uncertainty from the transient spikes the file flags in AdaNav. At a fixed rate of about 5% of states, chosen uniformly at random, we also query the teacher as an audit. We label the student "in error" at an audited state if the student's top action disagrees with the teacher's. We know the teacher is fallible here, and the experiments below check this label against downstream task failure. An isotonic calibrator is refit each round on recent audits with exponential down-weighting of old rounds, and it gives $\hat p(s)$, the estimated probability that the student errs at $s$. The teacher is called when

$$\hat p(s)\,V \;>\; c,$$

where $c$ is the teacher's expected token cost for one intervention and $V$ is the estimated value of a correction. We estimate $V$ from audited states as the difference in eventual task success between trajectories the teacher corrected and those it did not, expressed in the same token units through a fixed exchange rate that sweeps out a cost–success curve. This is DAW's step-function rule with costs attached to the two errors and with distributions refit as the student changes.

Once the teacher takes over, it continues while the student, run in parallel at negligible cost, would still choose a different action than the teacher, up to a cap $K_{\max}$. Control returns as soon as they agree. This makes the continuation length adaptive using a signal we get almost for free during the intervention. All teacher steps are kept, following OP-SHORT's finding that outcome filtering wastes tokens. The supervised loss covers only the teacher-generated steps, and GRPO handles the rest.

Baselines

The naive reference is plain behavioral cloning on offline teacher trajectories with the same teacher-token budget, followed by the same GRPO stage. It tells us whether on-policy intervention helps at all in our multi-round setting.

The strongest existing method is OP-SHORT with random switch times and the best fixed $K$ from a sweep over {1, 3, 5}. The sweep is charged to OP-SHORT's tuning budget, not ours. It is the direct competitor because it already beats behavioral cloning at matched supervision. The file reports no absolute success numbers for it: on Terminal-Bench-Dev the only figure is that OP-SHORT with 10% of the behavioral-cloning data matched 100% behavioral cloning plus reinforcement learning. We will therefore reproduce it ourselves. For the web benchmarks, the reported reference points are from Vattikonda et al.: 35.4% WorkArena success for supervised fine-tuning plus reinforcement learning, 28.4% for supervised fine-tuning alone, and 8.0% for reinforcement learning alone, plus 67.2% success on MiniWoB++. We will reproduce these under our budget before comparing.

The heuristic competitor is an AdaNav-style gate under two settings, each tuned once on round one and then frozen: a fixed absolute entropy threshold, and a fixed top-20% entropy percentile in the style of Prefix-RFT (Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling). We also include an OP-CRITICAL trigger, which calls the teacher only on explicit execution failures, and a TAPO-style collapsed-group trigger. Two ablations isolate our ingredients. The first keeps our adaptive stopping but uses a frozen, non-refit calibrator. The second keeps our calibrated trigger but uses a fixed $K$.

Every method gets the same student, teacher, environments, number of rounds, GRPO budget, and total teacher-token budget. Audit tokens count against our budget. Each method gets the same number of hyperparameter trials. We count our method as winning if its curve of success against teacher tokens dominates OP-SHORT and both entropy gates across budgets. The specific target is reaching OP-SHORT's best success with at most 60% of its teacher tokens, or beating it by more than seed variance at equal tokens, by round three or later.

Why not just…?

Why not just threshold entropy at a fixed percentile, as AdaNav and Prefix-RFT do? This is the real threat. AdaNav shows that raw entropy already identifies confusing steps well. Our bet rests on drift: as GRPO sharpens the policy, the entropy level that signals an error should shift. The audits let us test this directly. We will plot, per round, how well entropy alone separates error states from correct ones, and where the error and correct densities cross. If the crossing point is stable and the frozen percentile gate matches ours in later rounds, the calibration machinery is unnecessary. In that case we will report the simpler gate as the recommendation and keep only adaptive stopping as the contribution.

Why not just use random switching with a larger budget, as in OP-SHORT? Random switching spends teacher calls in proportion to trajectory length, not difficulty. AdaNav's finding that 71% of rollouts belong on the hardest trajectories suggests most random calls land on easy states. The cost–success curves settle this. If random switching matches us at every budget, then gating adds nothing.

Why not just spend the budget on more reinforcement learning after an early hand-off? Vattikonda et al. show early branching is compute-efficient. They also show that list-filtering tasks stay at 0% even after extensive supervised fine-tuning plus reinforcement learning, and that pure reinforcement learning reaches only 8.0% on WorkArena. We will match total compute, converting teacher tokens to student training steps at measured GPU cost. If more GRPO steps match our curve, the teacher calls are not worth their price.

Why not just call the teacher on explicit failures? OP-CRITICAL is cheap, but it fires only after the damage is visible. Many errors in multi-step tasks, such as a missed filter or a wrong search hop, do not throw an error until much later. A breakdown of where interventions land in successful corrections will show whether our gate catches errors earlier than failure signals do.

Proposed Experiments

We run on HotpotQA, ALFWorld, and Terminal-Bench-Dev, following OP-SHORT, and on MiniWoB++ and WorkArena through BrowserGym, following Vattikonda et al. Each method runs for four rounds with three seeds. The primary metric is the area under the curve of success rate against cumulative teacher tokens. Secondary metrics are success at fixed budgets, teacher calls per trajectory, mean realized $K$, and WorkArena per-category success with list-filtering tasks reported separately.

Four checks decide the outcome. The first is the per-round drift plot described above, which either supports or retires the calibration argument. The second is the label check. Using held-out audited states, we test whether teacher disagreement predicts eventual task failure. If it does not, we switch the gate's training label to outcome-based failure from rollout continuations, which costs more but is grounded in the task. The third tests adaptive stopping against fixed $K$ on HotpotQA, where non-monotone behavior in $K$ is documented. Adaptive stopping should recover or beat the best fixed $K$ without a sweep. The fourth is the WorkArena list-filtering split. Any nonzero success there under equal budget would be evidence that targeted teacher calls reach states that uniform scaling misses. A continued 0% would mark those tasks as a capability gap that intervention timing cannot fix.