Subgoal-Forked Rollouts for Critic-Free Hierarchical RL

Language-model agents that act over dozens of steps, such as clicking through a phone app or navigating a text-based household, are usually trained with reinforcement learning that returns a single pass/fail signal at the end of each attempt. Two problems follow. The agent cannot tell which of its fifty actions caused the failure. And the usual remedy, a learned "critic" network that estimates how promising each intermediate state is, roughly doubles GPU memory and destabilizes training for large models. The cheapest known fix, Group-in-Group Policy Optimization (GiGPO), avoids the critic. It finds moments where different attempts happened to pass through the exact same environment state and compares what happened next. That only works when states literally repeat. On screenshots, continuous states, or any observation with incidental noise, states never repeat, and the method falls back to judging whole trajectories.

We propose to stop hoping that states repeat and to make them repeat on purpose, at the points where it matters. The agent first writes a short language subgoal ("open the settings app"), then acts until it declares that subgoal done. During training we snapshot the environment at these subgoal boundaries and fork several continuations from each snapshot. Some forks keep the subgoal and resample only the low-level actions. Others resample the subgoal itself. Forks from one snapshot share an identical starting state by construction, so comparing their final pass/fail outcomes gives step-level credit to the planner and to the executor without a critic, a process reward model, or an oracle. Because forks reuse the shared prefix, including its already-generated tokens and cached computation, the method should also use fewer generated tokens per useful comparison than independent rollouts.

Core Claim

The claim is that hierarchical decomposition should serve as a device for choosing where to branch during rollout collection, rather than as an extra network or a fixed prompt. Branching at the agent's own subgoal boundaries yields exact comparison groups in any observation space. Those groups give separate, unbiased critic-free advantages for "which subgoal to pick" and "how to carry it out." We expect this to beat both exact-hash GiGPO and branching at fixed step intervals, and to do so at matched token and environment-step budgets.

For a node $n$ in the rollout tree (a snapshot at some subgoal boundary), let $\bar R(n)$ be the mean terminal reward of all completed episodes descending from $n$. For a child $c$ of parent $p$, the advantage is

$$A(c) = \bar R(c) - \frac{1}{|C(p)|}\sum_{c' \in C(p)} \bar R(c'),$$

where $C(p)$ is the set of siblings forked from $p$. The planner's subgoal tokens receive $A(c)$ when the siblings differ in subgoal. The executor's action tokens receive $A(c)$ when the siblings share a subgoal. A trajectory-level group term, as in standard GRPO, is added on top.

Relation to Prior Work

GiGPO supplies the core method we reuse, which is group-relative advantages at shared states computed from a batch, at under 0.002% extra training time. It also motivates the proposal: its reliance on exact state matching is the failure we remove. It has no explicit subgoals, and its hierarchy lives only in the advantage estimator.

K²-Agent shows that a planner/executor split matters a great deal on AndroidWorld: 76.1% success versus 35.3% for a flat model. However, it trains only the executor with RL. Its planner is revised through a prompt-reflection loop seeded from one human demonstration. We train both levels with RL from the same terminal reward.

HAL shows that language subgoals plus hindsight relabeling beat HIRO and Option-Critic under sparse reward. Its relabeling, however, requires a language oracle that can describe any state. We need no oracle, because subgoal quality is judged only by what its forks eventually achieve.

Two lines of work constrain the design. ReasonRAG shows that step-level supervision can match outcome RL with 18x less data, but it builds that supervision with expensive token-level tree search. WebArbiter gets step judgments from a generative 7B judge distilled from o3, which adds heavy inference cost. Our tree branches only at the few subgoal boundaries, not at every token or action, so its cost stays within a small constant factor of ordinary rollouts.

On the theory side, GHRL and HierQk(λ) explain why credit that jumps across subgoals helps under sparse reward. Both do so in tabular mazes with given structure or explicit value functions. We test whether this benefit survives when the value function is replaced by forked Monte Carlo comparisons.

Research Program

We will build one vision-language or language model that emits, in the Chain-of-Action style of OpenHA, a subgoal string followed by primitive actions and an explicit end-of-subgoal token. The executor returns control after declaring the subgoal done or after a step cap. Using a single model avoids hosting separate planner and executor weights and the moving-target problem that comes with them.

Rollout collection works as follows. From each task we run a root episode. At each of the first $d$ subgoal boundaries, we snapshot the environment and fork $b$ children. Half of the children resample the subgoal and half keep it. We reuse the model's cached context for the shared prefix. Environment snapshots are cheap in ALFWorld, WebShop, and Sokoban, which are text or grid simulators, and are supported by emulator snapshots in AndroidWorld. We will measure the snapshot cost directly rather than assume it.

The analysis has three parts. The first is advantage variance per generated token, compared across forking, GiGPO, and GRPO. The second is how often each method produces a non-degenerate step-level group, meaning one that is not all-success or all-failure. The third is whether planner advantages concentrate on the subgoals that humans would identify as decisive.

Baselines

The naive reference is trajectory-level GRPO with the same model and terminal reward: independent rollouts, one advantage per episode, no hierarchy. It fixes the floor. It is also the method GiGPO reports beating by more than 12% on ALFWorld and more than 9% on WebShop. The file gives no absolute ALFWorld or WebShop success values for either method, only these relative gains, so we will re-run both ourselves.

The strongest existing critic-free competitor is GiGPO itself, run with its exact-hash anchor states. It reports 81.0% success on Sokoban in the VLM setting (GiGPO). On AndroidWorld, the strongest reported hierarchical system is K²-Agent at 76.1% success (K²-Agent). We will run K²-Agent-style training, meaning C-GRPO on the executor with a prompted planner, under our backbone and budget. That makes it the hierarchical competitor that does not train the planner.

The comparison will be fair along four axes. All methods use the same backbone and the same number of environment steps and generated tokens per update; for forking, tokens are counted with prefix reuse credited only as actually saved. All methods get the same hyperparameter sweep size and three seeds. Forking methods receive no extra wall-clock time.

Two ablations isolate the key ingredient. Fixed-interval forking branches every $k$ actions, with $k$ matched to the mean subgoal length, and uses no subgoals. It tests whether subgoal boundaries matter or whether any branching suffices. Subgoal-free forking, where the model emits no subgoal text but forks at the same times, tests whether the language subgoal matters beyond marking where to branch.

We count as beating GiGPO a statistically reliable gain on Sokoban and ALFWorld at matched tokens. On the perturbed-observation variants described below, we expect a gap that widens as exact-hash matching degrades. We count as beating the K²-Agent-style baseline a gain on AndroidWorld at equal compute, which would show that RL on the planner adds something beyond prompt reflection.

Why not just…?

Why not just make GiGPO match states by embedding similarity instead of exact hashes? This is the most serious threat, and we run it as a baseline. We cluster observation embeddings with a threshold swept over the tuning budget and apply GiGPO's grouping to the clusters. The objection to it is bias. Two "similar" screenshots can differ in exactly the detail that decides success, such as a toggle's state, so the comparison can credit the wrong action. Forked states are identical, and the resulting comparison is unbiased. On the perturbed ALFWorld variant, where the true state is known, we can measure how often embedding clusters merge states whose true values differ. If embedding matching equals forking at lower cost, we will report that, and the contribution shrinks to the planner-credit component.

Why not just fork at fixed step intervals without any hierarchy? This is the second real threat, and it is the ablation described above. Our hypothesis is that mid-subgoal forks mostly compare near-identical low-level continuations that share an outcome. Such groups are all-success or all-failure and so carry no signal, whereas boundary forks split exactly where outcomes diverge. The non-degenerate-group rate settles this directly. If fixed-interval forking matches ours, the hierarchy is not pulling its weight and we will say so.

Why not just keep the planner as a prompted model and RL only the executor, as K²-Agent does? K²-Agent's planner needs a human demonstration and a reflection loop, and it cannot improve from reward. Its reported hierarchical gain comes with a planner that never receives gradients. Our AndroidWorld comparison, and an ALFWorld variant with no demonstration, test whether reward-driven planner updates add success on top of executor RL.

Why not just train a process reward model or run tree search, as ReasonRAG and WebArbiter do? Both deliver step-level signal, but at the cost of token-level search or a large generative judge. WebArbiter additionally depends on distillation from a proprietary teacher. We report success per generated token. If a process-reward approach wins on success but loses heavily on tokens, the trade-off is exactly the one this proposal targets.

Proposed Experiments

We will train on ALFWorld, WebShop, and Sokoban, which are GiGPO's settings, and on AndroidWorld with screenshots only, which is K²-Agent's setting. To test the core claim directly, we will build perturbed ALFWorld and Sokoban variants that inject task-irrelevant observation noise: randomized distractor lines, pixel jitter, and irrelevant timestamps. These change what the agent sees but not what it should do. Exact hashing should collapse on them while forking should not.

The metrics are task success rate, success as a function of generated tokens and of environment steps, peak GPU memory, the non-degenerate step-group rate, and the variance of the advantage estimate.

Three checks are decisive. First, as noise increases on the perturbed variants, GiGPO's step-group rate should fall toward zero and its success toward GRPO's, while subgoal forking should stay flat. Second, at matched tokens, subgoal forking should beat fixed-interval forking. If it does not, the hierarchy's role is refuted. Third, on AndroidWorld, adding planner advantages to executor-only training should raise success. If none of the three effects appears, we will conclude that critic-free hierarchical credit needs something other than exact shared starting points.