A robot or software agent that follows a plan constantly sees small mismatches between what it expected and what happened. A block lands a few millimetres off, a grasp slips once, a web page shows a different advertisement. Some mismatches mean the plan is wrong. Some mean the agent's model of the world is missing something it could learn. Some are plain randomness that no amount of data will remove.
Current closed-loop agents treat all three the same way: any large mismatch triggers replanning. Current active learners, the algorithms that choose which experience to collect next, make the mirror-image mistake and chase the most surprising outcomes, which are often the noisiest ones. We propose a single runtime check that sorts each surprise into one of three bins. Surprises that the agent's own noise model already explains are ignored. Surprises the agent cannot explain but has seen often enough cause immediate replanning. Surprises in regions where it has little data are kept as priority training data for its model of which plan steps succeed.
The claim is that this sorting cuts wasted replans and wasted learning interactions together. It should matter most when noise is uneven across the task, which is the common case in physical manipulation. A second goal is to explain a known puzzle. Separating the two kinds of uncertainty barely helps in standard pool-based image labelling, yet appears to help a great deal in reinforcement learning. We will test whether the difference comes from uneven noise combined with the agent choosing its own data.
Core Claim
Two pieces of the literature motivate this. Closed-loop planners such as ATLAS halt and replan whenever an observed outcome departs from a look-ahead prediction. That design gave ATLAS 63.0% success on WebArena-Lite, but by its own construction it cannot tell a misfiring trigger caused by page randomness from a real plan failure. When Replanning Becomes the Bottleneck shows what over-triggering costs. Unbudgeted replanners violate their deadlines 85.5–100% of the time. Its fix, BRACE, uses fixed cooldown and commit windows that ignore why a trigger fired.
On the learning side, Active Learning of Abstract Plan Feasibility learns which multi-step block-stacking plans will work in 200–400 interactions by querying plans on which an ensemble of graph networks disagrees. It does this offline, before execution, and it never separates stochastic outcomes (a tower that sometimes stands) from structural ones (a tower that never can).
Two further results constrain and supply the method. The Interplay of Uncertainty Modeling and Deep Active Learning is a constraining negative result: in static image-classification pools, separating "the model doesn't know" (epistemic uncertainty) from "the outcome is random" (aleatoric uncertainty) gave negligible gains over plain entropy. Modelling the noise more faithfully during selection, for example with Mixup, actually hurt. Disentangling Epistemic and Aleatoric Uncertainty in RL and Taming Aleatoric Impulse supply the method. They show that sampling-free epistemic estimators (deep kernel learning and posterior networks, the latter an evidence-counting density head) detect unfamiliar inputs at AUC-ROC near 1.0, where dropout and ensembles sit near 0.5. They also show that damping early noise-driven value spikes matters in sequential control.
Our hypothesis is that decomposition pays off only when two conditions hold together. The first is that noise is heteroscedastic, meaning concentrated in particular regions of the task. The second is that the agent's own decisions determine which regions it visits, so chasing noise compounds over time. When either condition is removed, we predict the gain disappears, which would reproduce the image-classification null result.
Research Program
The agent keeps a step-outcome model that, for each plan step in context, predicts a distribution over outcomes, including success or failure and key state features. The model has two outputs. The first is a distributional head whose spread estimates irreducible noise at that step. The second is an evidential density head that estimates how much relevant data the model has seen.
At execution time the agent compares the actual outcome with the prediction. Let $s_t = -\log p(o_t \mid x_t)$ be the surprise of outcome $o_t$ given step context $x_t$. Let $h_a(x_t)$ be the expected surprise implied by the noise head alone, and let $u_e(x_t)$ be the epistemic score. The triage signal is the excess surprise
$$ \Delta_t = s_t - h_a(x_t). $$
The rule then works as follows. If $\Delta_t$ is small, the agent continues, because the noise model already explains the outcome. If $\Delta_t$ is large and $u_e$ is low, the agent replans at once and overrides any cooldown window, because a well-understood step failing beyond its noise level signals a structural problem. If $\Delta_t$ is large and $u_e$ is high, the transition goes into a prioritised buffer for updating the outcome model, and the agent replans conservatively.
Plan selection during learning episodes scores candidate plans by expected epistemic reduction minus a penalty on predicted noise. This is the same kind of noise penalty that Taming Aleatoric Impulse uses on critics, here moved to the level of plan choice. The candidate pool reuses the infeasible-subsequence pruning from the plan-feasibility paper.
The work has three parts. The first is building the triage module for a simulated block-stacking domain and for a large-language-model replanner. The second is a controlled study that varies noise heterogeneity and whether the agent chooses its own data, to locate the predicted crossover where decomposition starts to help. The third is checking that each of the three branches does what it claims. Ignored surprises should have no effect on later task success. Override replans should be followed by recovery. Buffered transitions should reduce held-out prediction error per sample faster than random transitions do.
Baselines
The naive reference point has two versions. One executes plans open-loop, with no monitoring. The other replans on any discrepancy above a fixed threshold, which is the ATLAS-style trigger. Both run with the same planner, outcome-model architecture, and interaction budget as our method.
The strongest replanning competitor is BRACE from When Replanning Becomes the Bottleneck. On RoboFactory it reports 80% task success with 4.6% deadline violations, against 0% for unbudgeted replanning. We will tune BRACE's cooldown and commit windows with the same hyperparameter-search budget we give our thresholds. Beating it means matching or exceeding its success rate at equal or lower violation rates, with the gap growing as injected noise becomes more uneven. Matching BRACE only at uniform noise would be an acceptable and informative result.
The strongest learning competitor is ensemble-disagreement plan querying (BALD) from Active Learning of Abstract Plan Feasibility. It reports peak simulated feasibility performance within 200–400 interactions, against more than 800 for greedy selection. It also reports 90% success on its "Longest Overhang" task, against 20% for an analytical model. We rerun it in the same simulator with the same graph-network backbone and the same number of ensemble members' worth of compute. Beating it means reaching its peak accuracy in fewer interactions once stochastic block outcomes are injected, while not losing to it in the noise-free original setting.
We also include two cheaper uncertainty selectors as baselines. One uses total predictive entropy. The other is γ-Progress from Active World Model Learning with Progress Curiosity, which rewards how fast model loss is falling and was 13.79x better than random in its "Noise World."
Ablations isolate the key ingredient. "Gate-only" keeps triage but never learns online. "Learn-only" learns from all failures but replans on raw discrepancy. "Collapsed" replaces $\Delta_t$ with raw surprise $s_t$, which tests whether the noise head itself carries the benefit.
Why not just…?
Why not just use cooldown and commit windows? BRACE's results show these work remarkably well. That makes them a real threat on the churn question. They are blind to cause, though. A dropped tool in the first step of a commit window still forces the agent through the remaining actions, and BRACE's authors list risk-aware budgeting as open. Our override branch targets exactly this case. The heterogeneity sweep will show whether the advantage is real or confined to rare events. If BRACE matches us across the sweep, the gating half of the proposal is not worth its cost.
Why not just retry the failed step and see if it fails again? Retrying is the most dangerous alternative, because repeated trials directly estimate noise. Its cost is the extra interactions and time spent at every surprising step, and it learns nothing that transfers to similar steps elsewhere. Our noise head is effectively an amortised retry model. We include retry-once and retry-twice policies under the same interaction budget. If they match our sample efficiency, we will report that amortisation does not pay in these domains.
Why not just use total entropy, or learning progress, which avoid explicit decomposition? In static pools entropy is as good as decomposed scores, per The Interplay of Uncertainty Modeling and Deep Active Learning, so our own uniform-noise, fixed-pool condition should reproduce that null result. γ-Progress already avoids white noise, but it needs repeated visits to measure progress and gives no instantaneous signal for whether to replan. The crossover study tests directly whether decomposition beats both selectors only when noise is uneven and the agent chooses its own data.
Why not just pre-train in simulation? The plan-feasibility paper's 90%-versus-20% gap between learned and analytical models argues against relying on priors alone. Our method still starts from a simulator-pretrained outcome model, so this alternative is complementary rather than competing.
Proposed Experiments
The primary testbed is a PyBullet-style re-creation of the plan-feasibility block-stacking setting, using cuboids with hidden weights. We inject gripper slip and placement jitter whose magnitude depends on block geometry, at five heterogeneity levels ranging from uniform to concentrated in a few block types. Metrics are interactions needed to reach a fixed feasibility accuracy, downstream "Longest Overhang"-style task success, and the fraction of queried plans whose outcomes are mostly random.
The second testbed is RoboFactory with the BRACE replanner. We add perception jitter and transient contact failures, and measure success, deadline-violation rate, replans per episode, and tokens per episode. We expect fewer tokens because replans that the noise model explains are suppressed. A transfer check on WebArena-Lite adds nondeterministic advertisements and layout shifts to pages and compares ATLAS's discrepancy trigger with our gate. The reference point is ATLAS's reported 63.0% without perturbation. We report no claim about perturbed numbers in advance, since none exist.
The decisive check is the two-factor study: noise heterogeneity crossed with whether the agent chooses its own data versus drawing from a fixed pool. The hypothesis is supported only if decomposition's advantage over entropy is near zero in the uniform fixed-pool cell and grows with both factors. If the advantage is flat everywhere, the image-classification null result extends to closed-loop agents, and we will say so.
References
- ATLAS: Actor-Critic Task-Completion with Look-ahead Action Simulation
- When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agents
- Active Learning of Abstract Plan Feasibility
- The Interplay of Uncertainty Modeling and Deep Active Learning: An Empirical Analysis in Image Classification
- Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning
- Taming Aleatoric Impulse in Off-Policy Reinforcement Learning
- Active World Model Learning with Progress Curiosity