Many robot systems now split control into two layers. A slow, large generative model imagines where the robot should be a few seconds from now, and a fast, small policy drives the motors to get there. The split works because the slow model brings broad knowledge and the fast model brings reactivity. Its weak point is that the slow model has no idea what the fast model can actually do. It will happily propose a subgoal image with the gripper inside the table, or an arm pose the controller cannot reach. We propose to close this gap with a signal the system already learns for free. A low-level policy trained to reach goals, without any task reward, can also be trained to estimate whether it can reach a given goal from the current state, and whether it can get there without a violation. We use that estimate, a goal-conditioned value function, to steer the high-level generator's sampling during the final denoising steps only. The result is subgoals the controller can achieve and achieve safely, with no analytic barrier function and no hand-weighted reward or cost terms.
The core claim is that the low-level controller's own learned value function can replace the hand-specified constraints that current safe generative planners assume, and can fix the open-loop subgoal problem in hierarchical systems. It does both only if it is applied as a late-stage correction rather than throughout sampling. Applied throughout, it causes the same planning failures that safety guidance in intermediate steps is known to produce.
Where this sits
The hierarchical side motivates the problem. SuSIE finetunes an image-editing diffusion model (a generator that produces images by gradually removing noise) to produce future subgoal images, which a separate diffusion policy then reaches. Its authors note that the subgoal generator "doesn't know what the low-level policy is actually capable of." UniPi generates whole task videos and decodes them with an inverse dynamics model, and HiRT passes latent features from a slow vision-language model to a 10 Hz policy. None of the three feeds anything about the controller's limits back to the planner, and none enforces safety.
The generative-safety side supplies the method and also limits it. CoDiG adds a barrier-function gradient to reverse diffusion and hands the result to a tracking model-predictive controller. Constrained Diffusers casts sampling as constrained optimization over discrete control barrier functions, which are hand-derived functions that are positive exactly on the safe set. PolyFlow keeps flow-matching iterates inside a convex polytope by construction. All of them need the constraint written down analytically, in a low-dimensional state space. We drop that assumption and replace the constraint with a learned quantity that also works on image goals.
SafeFlowMatcher constrains our design. It reports that pushing intermediate samples away from obstacles produces a 70–100% trap rate for SafeDiffuser in mazes, against 0% when the method first predicts unconstrained and only then corrects. We adopt that prediction-then-correction schedule.
Two papers come closest to our idea. Diffusion-based Planning with Learned Viability Filters already uses learned critics to reject candidate trajectories with delayed failures. FISOR already learns feasible sets from offline data using Hamilton-Jacobi reachability, a method for computing the set of states from which safety can be maintained. Our proposal differs from both in three ways. The critic is the low-level controller's own reach-and-safety value. It acts on the interface between the two layers rather than on a single flat trajectory planner. And it uses gradient guidance in addition to filtering, which we test head-to-head against filtering.
On the reward side, the low-level value is learned by relabeling reached states as goals, in the spirit of self-supervised goal-conditioned shaping. The only human input is a binary violation event such as contact with a forbidden region, never a weighted sum of terms. This removes the trade-off tuning that FISOR and DRCORL still inherit from pre-engineered reward and cost signals.
Research Program
The work builds three components. The first is a goal-conditioned low-level policy trained by hindsight relabeling on play data. The second is two value heads attached to that policy. $V_{\text{reach}}(s,g)$ estimates the discounted probability that the policy reaches goal $g$ from state $s$, and $V_{\text{safe}}(s,g)$ estimates the probability that it does so with no violation event. The safety head is trained with the reversed expectile regression used by FISOR, so that it is pessimistic. The third is a high-level subgoal generator, either image-space as in SuSIE or state-space, whose sampling is corrected with
$$\tilde{\nabla}_g \log p_t(g \mid s,\ell) = \nabla_g \log p_t(g \mid s,\ell) + \lambda_t \nabla_g \big[\log \underline{V}_{\text{reach}}(s,g) + \log \underline{V}_{\text{safe}}(s,g)\big].$$
Here $p_t$ is the generator's distribution over subgoals $g$ at noise level $t$, given state $s$ and instruction $\ell$. The terms $\underline{V}$ are the minimum over a five-member ensemble, so that guidance cannot climb into regions where one critic is wrongly optimistic. The weight $\lambda_t$ is zero for all but the last fraction $\tau$ of denoising steps.
After sampling, a subgoal whose $\underline{V}_{\text{safe}}$ falls below a threshold $\delta$ is resampled. At execution time the low-level policy replans whenever a new subgoal arrives.
The analysis focuses on one failure mode, which is the reward-hacking problem in a new form. Guided sampling may find subgoals that fool the critic. We measure calibration directly by comparing predicted $V_{\text{reach}}$ against the reach rate actually achieved, separately for in-distribution and guided subgoals. This gap is the quantity that tells us whether the learned certificate can be trusted.
Baselines
The naive reference is unguided SuSIE-style generation with the same low-level policy. On CALVIN, SuSIE reports an average of 2.69 consecutive instructions completed in the multi-environment zero-shot test, against 1.43 for AugLC. We retrain it with our low-level policy so that only the subgoal selection differs. Beating it means a longer completion chain together with fewer violations.
The strongest competitor is best-of-N filtering with the same ensemble critic. It draws N unguided subgoals and keeps the one with the highest $\underline{V}_{\text{reach}} \cdot \underline{V}_{\text{safe}}$, which is the viability-filter recipe of Learned Viability Filters applied at the layer interface. It is strongest because it uses the same information as our method and never pushes samples off the generator's distribution. We match wall-clock time per subgoal on the same GPU by giving best-of-N as many samples as our guided method's gradient steps cost. No reported result was found on Lacuna for this configuration. Beating it means a statistically significant gain at matched compute.
A low-level-only safety reference keeps the planner unguided and applies a QP-CBF filter from CBFkit to the motor commands. This is possible only in the state-space tasks, where an analytic barrier exists.
An oracle reference uses CoDiG-style guidance with a hand-engineered cost on the state-space tasks. It shows how much is lost by learning the constraint instead of writing it down. On Maze2D, SafeFlowMatcher reports a 0% trap rate and 100% safety with analytic barriers, which sets the ceiling.
Two ablations isolate the key ingredient. The first applies guidance at every step ($\tau=1$) against late-only. The second keeps only the reach head or only the safety head. All methods share data, the low-level policy, and a fixed tuning budget of twenty configurations each.
Why not just…?
Why not just use best-of-N rejection with the critic? This is a real threat, and we take it seriously. If feasible subgoals are common among unguided samples, filtering should match guidance at lower complexity. In that case our contribution reduces to the observation that the controller's own value function is the right critic. That would still be useful, but it is a smaller result. We settle the question by creating environments where the fraction of feasible subgoals is controlled, using clutter density and forbidden-zone size. We predict guidance wins only when that fraction falls low enough that N samples rarely contain a good one. If the curves never separate, we report filtering as the method.
Why not just put a barrier-function safety filter on the motors and leave the planner alone? SafeFlowMatcher shows that constraint-blind references followed by local correction produce 70–100% trap rates in mazes. CoDiG argues that its tracking controller needs references that already steer around obstacles. Beyond that evidence, an analytic barrier does not exist for image subgoals in CALVIN. The state-space comparison measures the trap rate directly.
Why not just finetune the subgoal generator on subgoals the controller actually reached? Hindsight finetuning teaches the generator average reachability, but it cannot respond at test time to a new obstacle. This is the same limitation PACT accepts in exchange for zero runtime cost. We include the finetuned generator as an additional baseline and evaluate it on obstacle layouts held out from finetuning, where we expect it to degrade while guidance does not.
Proposed Experiments
The first stage is state-space. It uses Maze2D and the Gym locomotion overhead-obstacle tasks from Constrained Diffusers, which report 0.09% violations with the augmented Lagrangian method against 4.82% for unconstrained diffusers. Here we replace their inverse dynamics model with our learned goal-reaching policy and measure trap rate, violation rate, and return against the oracle analytic-barrier methods.
The second stage is CALVIN with image subgoals. We add forbidden table regions and fragile objects whose contact counts as a violation, and measure consecutive instructions completed, violations per episode, critic calibration, and subgoal-generation latency.
Three checks decide the outcome. First, does late-only guidance beat matched-compute best-of-N in the low-feasibility regime? Second, does every-step guidance reproduce the trap-rate failure that SafeFlowMatcher describes? Third, does the gap between predicted and achieved reach rate stay small on guided subgoals? If it does not, the method is exploiting its critic, and its safety claims fail regardless of headline success rates.
References
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
- Learning Universal Policies via Text-Guided Video Generation
- HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
- Constraint-Aware Diffusion Guidance for Imitation Learning
- Constrained Diffusers for Safe Planning and Control
- PolyFlow: Safe and Efficient Polytope-Constrained Flow Matching with Constraint Embedding and Projection-free Update
- SafeFlowMatcher: Safe and Fast Planning using Flow Matching with Control Barrier Functions
- Diffusion-based Planning with Learned Viability Filters
- Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
- Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL
- CBFkit: A Control Barrier Function Toolbox for Robotics Applications
- PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulation