Branching Groups from Shared Robot States: Spending Scarce Rollouts Where Outcomes Diverge

Reinforcement learning can push a robot policy past the quality of the human demonstrations it was trained on. On long, multi-stage tasks, though, the most popular critic-free method wastes most of the rollouts it collects. That method is group relative policy optimization (GRPO), which scores each attempt against a group of sibling attempts rather than against a learned value estimate. It runs several complete episodes from the same starting scene and rewards the ones that beat the group average. In a kitchen task where an early-training policy almost never finishes, every episode in the group fails, the group average equals every score, and the update is zero. The robot has paid for full-length rollouts and learned nothing. Robotics is a field where each rollout is expensive and demonstrations are scarce, so this is exactly the wrong place to waste samples.

We propose to change where the groups start. Instead of sampling every group from the initial scene, the method picks intermediate physical states: states the policy reached on its own, or states taken from the handful of available demonstrations. It chooses only those where a cheap two-rollout probe gives disagreeing outcomes. It then runs several short continuations from each chosen state and computes group-relative advantages among them. Each group then shares an exact physical "prefix", success and failure are both likely inside it, and the binary task-success signal is enough. No hand-crafted quality score, milestone detector or learned critic is needed. The claim is that, at a fixed budget of environment steps, this state-branched GRPO learns long-horizon manipulation substantially faster than GRPO from the initial state. The claim also covers GRPO with dense subtask rewards and GRPO with execution-quality splitting.

Core Claim

Two results from language-model reasoning motivate the proposal. The first is GRPO is Secretly a Process Reward Model, which shows that when rollouts in a group share a prefix, GRPO's advantage acts as a Monte Carlo estimate of how good that prefix is. The second is Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO, which proves that adding more continuations $M$ per shared prefix shrinks advantage-estimation variance at rate $O(1/M)$. Adding more independent prefixes, by contrast, leaves a variance floor. In text, prefixes are token strings. In a simulator, a prefix can be a saved physical state that is restored exactly. That turns tree-style branching into a tool for robot learning, as long as one decides which states to branch from.

Our selection rule rests on a simple fact. With binary success rewards, a group from a root state $s$ is informative only if its continuations disagree. The chance that $M$ continuations all agree is $p(s)^M + (1-p(s))^M$, where $p(s)$ is the policy's success probability from $s$. That chance is smallest when $p(s)\approx 0.5$. We therefore probe each candidate root with two continuations. We keep roots whose two outcomes differ, and those two probe rollouts become part of the group, so probing costs nothing extra on the roots kept. For each kept root we add $M-2$ further continuations with rewards $r_1,\dots,r_M \in \{0,1\}$ and compute

$$A_j = \frac{r_j - \bar r(s)}{\operatorname{std}(r(s)) + \epsilon},$$

where $\bar r(s)$ and $\operatorname{std}(r(s))$ are the mean and standard deviation over the $M$ continuations from root $s$, and $\epsilon$ is a small constant. The policy-gradient loss uses only the actions taken after $s$. This keeps credit local to the decisions that actually changed the outcome. It also avoids the problem Sullivan and Koller call popularity bias, in which a shared prefix is over-penalized in proportion to how many rollouts contain it.

Relation to Existing Bridges

Several papers already apply group-relative RL to embodied control, and each leaves the long-horizon, low-data regime open in a different way.

Prism-GRPO attacks the same all-fail and all-succeed group problem on RoboTwin 2.0. It adds a continuous execution-quality term to the reward, built from signals such as joint reversals, contact forces on non-target objects, or checks by a vision-language model. Those signals must be engineered per task. We keep the reward binary and change where groups start, so the two ideas can be combined. Learn Where Outcomes Diverge (PCM) finds where success and failure diverge along the trajectory, but only to drop low-variance chunks from the backward pass. It saves compute on rollouts that were already collected and relies on gripper-state heuristics to define phases. We use divergence to decide which rollouts to collect.

Milestone-Guided Policy Learning (BEACON) shows that computing advantages against trajectories that reached the same milestone nearly doubles ALFWorld success over GRPO. Its milestone detector is engineered by hand, and it has not been tested on physical manipulation. Our divergence probe takes the place of that detector. GRPO-MA does report a simulated manipulation setting (ManipLLM on PartNet Mobility). Its branch points, however, are textual thought delimiters, not physical states in a closed-loop multi-stage task.

The remaining bridges constrain the design rather than compete with it. Constrained GRPO and Plan-R1 show that dividing a mixture of reward components by the group standard deviation distorts the low-variance component. That is one reason we avoid mixed shaped rewards. Rank-Then-Act trains its video ranker with GRPO but trains the policy itself with PPO or DrQv2.

On the data-scarcity side, RoboCasa motivates the problem. Even with thousands of synthetic demonstrations, composite long-horizon tasks stay "often below 12%" success. Adding more imitation data does not fix multi-stage compounding error.

Research Program

We build the method on a vision-language-action policy (a model that maps camera images and a language instruction to robot actions) with a tokenized action head. Per-action log-probabilities are then exact, and GRPO's likelihood ratio is well defined. The policy is first trained by supervised imitation on a deliberately small demonstration set, for example the 50 human demonstrations per task that RoboCasa uses as its low-data point.

During RL, each iteration draws candidate roots from two pools. One pool is states along the current policy's own rollouts. The other is states along the scarce demonstrations, restored in simulation. The demonstration pool is how the method uses scarce data a second time. Demonstrations are normally consumed once, for imitation. Here they also serve as launch points into late task stages the policy cannot yet reach, which gives RL signal on the final stages from the first iteration.

Each iteration then runs the probe-and-keep rule on the candidate roots, completes the kept groups, and applies the per-root advantage above with a clipped policy-gradient update.

The analysis has two parts. The first extends the GRPO-MA variance argument from token prefixes to restored physical states. We will predict the fraction of non-degenerate groups as a function of $p(s)$ and $M$, and check that prediction against the fraction measured during training.

The second part examines how the root pools shift. States from the demonstration pool are off the policy's own state distribution. We will measure whether skills learned from those roots transfer to states the policy reaches itself. We will also measure whether the fraction of roots drawn from demonstrations should be annealed toward zero as training progresses.

A secondary track tests a flow-matching action head, using the chunk-level likelihood surrogate from Reinforcement Learning for Flow-Matching Policies. That paper reports GRPO incurring 50% to 85% less cost than imitation-only flow matching, but only on unicycle dynamics.

Baselines

The naive reference point is the supervised policy trained on the same small demonstration set, with no RL. For RoboCasa composite tasks the relevant published number is the imitation ceiling of "often below 12%" success (RoboCasa). On LIBERO, RedFlow reports a base-to-corrected LIBERO average of 56.2% to 68.2% with offline redirection. Every RL method starts from our own supervised checkpoint, so the gain over it is measured directly.

The first RL baseline is vanilla GRPO from the initial scene with binary success reward. This is the method we claim is wasteful. It gets the same total number of environment steps; we count steps, not episodes, because short continuations are cheaper than full episodes. It also gets the same group size and the same learning-rate sweep. Beating it means reaching higher success at every matched step budget, together with a higher fraction of non-degenerate groups. BEACON's language-agent analogue of that fraction rose from 23.7% to 82.0%.

The strongest existing method on the degenerate-group question is Prism-GRPO. On RoboTwin 2.0 Lift Pot it reports saving "56% of rollouts (roughly 20,500 trials)" relative to binary GRPO, and an execution-quality score of 0.82 versus 0.21 at 40k rollouts. We run it on RoboTwin 2.0 with the quality signals it describes, under our step budget. On RoboCasa and LIBERO we implement its success-plus-0.2-times-quality reward, using a comparable joint-reversal quality term. Beating it means matching or exceeding its rollout savings on its own RoboTwin tasks with binary reward alone. It also means clearly exceeding it on multi-stage tasks, where a whole-trajectory quality score cannot tell a late-stage failure from an early one.

For compute rather than sample efficiency we compare against PCM. PCM reports a "2.38x wall-clock speedup" to 98% of target success on LIBERO. We also combine PCM with our method, since the two touch different stages of training.

Two ablations isolate the key ingredient. The first keeps state branching but picks roots uniformly at random, without the disagreement probe; this separates branching from targeting divergence. The second draws roots only from the policy's own rollouts, with no demonstration pool; this measures how much the scarce demonstrations contribute as launch points. A third, smaller sweep reallocates a fixed budget between the number of roots and the number of continuations $M$, to test the predicted $O(1/M)$ advantage.

Why not just…?

Why not just give vanilla GRPO a dense reward that counts completed subtasks? Simulators such as RoboCasa expose subtask success checks, and a staged reward also breaks all-fail groups. This is the most serious threat, so we run it as a baseline under matched steps. There are two reasons to expect it to fall short. First, whole-episode groups still spend most of their steps replaying early stages the policy has already mastered, and the advantage is spread over the entire trajectory. Second, summing stage rewards and dividing by one group standard deviation is the scalarization that Constrained GRPO proves distorts low-variance components. If dense-reward GRPO matches us at equal steps, the contribution shrinks to "use subtask rewards", and we will report that. We will also combine the two, using stage checks to pick roots, to see whether they stack.

Why not just use a critic, as in PPO, to get state-level baselines? A value function also gives per-state baselines. For large vision-language-action models, though, it adds memory and must stay calibrated across tasks of very different difficulty. The backward pass already dominates cost, at roughly 78% of training time according to PCM. As evidence on sample cost, RedFlow reports on-policy baselines needing 13,000 to 24,000 trajectories to reach 75.8% on LIBERO-Spatial. We still run PPO with a value head on the same backbone and step budget. Our Monte Carlo baseline per root is unbiased and needs no learning, so the comparison asks whether that exactness beats a learned critic's ability to generalize across states.

Why not just collect or synthesize more demonstrations? RoboCasa's own scaling study answers much of this: 3,000 synthetic demonstrations lift atomic tasks from 28.8% to 47.6%, yet composite tasks stay below 12%. We will still include a supervised baseline trained on MimicGen-style amplified data. The experiment asks whether RL on 50 demonstrations beats imitation on many more.

Why not just reset to demonstration states with ordinary GRPO? This is essentially a reverse-curriculum start. It is covered by our random-root ablation restricted to demonstration states. If the ablation performs as well as the full method, the disagreement probe is unnecessary, and we would report that simplification.

Proposed Experiments

The main benchmarks are RoboCasa composite tasks, starting from 50 demonstrations per task, and LIBERO-Long, the long-horizon suite of the LIBERO benchmark. RoboTwin 2.0 serves as a head-to-head venue where Prism-GRPO's published numbers apply, and LIBERO-Spatial, -Object and -Goal serve as short-horizon controls. On the short-horizon controls we expect our advantage to shrink, because whole-episode groups are already informative there.

Metrics are success rate as a function of total environment steps, the fraction of non-degenerate groups, the variance of the policy gradient across seeds, and wall-clock time. We use five seeds per condition.

The decisive checks are as follows. The first is whether state-branched GRPO beats both vanilla GRPO and dense-reward GRPO on RoboCasa composite and LIBERO-Long at matched steps. The second is whether it matches Prism-GRPO's rollout savings on RoboTwin without any quality signal. The third is whether removing the disagreement probe erases most of the gain.

A final check addresses the main limitation, which is that exact state restoration only exists in simulation. We transfer the resulting policies into LIBERO-Plus, which perturbs camera viewpoint and initial state. There we test whether policies trained from restored roots are as robust as policies trained from full episodes. ACoT-VLA gives a reference point of 84.1% average success on LIBERO-Plus.