Branching Where Outcomes Diverge: Group-Relative RL for Few-Demonstration Robot Policies

Robot policies trained by imitation from a few dozen human demonstrations break down on long tasks. They drift off the demonstrated path, have never seen a recovery, and fail. Reinforcement learning could repair this, and one algorithm has become the default for fine-tuning large generative models: Group Relative Policy Optimization (GRPO). GRPO samples several attempts at the same problem and scores each attempt against the group average, so it needs no separate value-predicting network. On long robot tasks it stalls, for a simple reason. When the starting policy fails from almost every initial scene, every attempt in a group earns zero reward, the group has no spread, and the update is zero. The proposal is to build groups somewhere else. Instead of sampling many full attempts from the start of the episode, we reset the simulator to intermediate states taken from the demonstrations and from the policy's own failed runs. From each such state we branch a small group of alternative next actions and let the policy finish the episode. We choose branch points adaptively, preferring the states where the policy currently succeeds about half the time, because that is where a group's outcomes actually differ.

The expected payoff is twofold. Learning concentrates at the moments that decide success, such as the instant a grasp slips or an alignment goes wrong. The environment interaction needed to lift a 50-demonstration policy should also drop sharply, and interaction is the second scarce resource in robot learning after demonstrations. The regime is simulation (LIBERO, MimicGen, RoboCasa), where resets are cheap. In that setting, demonstration scarcity rather than simulator access is the bottleneck.

Core Claim

For a flow-matching visuomotor policy (a generator that turns noise into an action chunk by following a learned velocity field), we claim three things. Group-relative updates computed on short branches from variance-seeking intermediate states avoid the advantage collapse that stops standard GRPO on long-horizon manipulation. They assign credit to individual decision points without a learned critic or heuristic failure clustering. They reach higher success per environment trajectory than on-policy RL, offline correction methods, and world-model-based GRPO.

Concretely, let $s$ be a stored state and $\hat p(s)$ the policy's running estimate of success when continuing from $s$. We sample branch points with probability proportional to $\hat p(s)(1-\hat p(s))$. This is the variance of a binary success outcome, so it is also the expected spread in a group's rewards, and it is maximal at $\hat p = 0.5$. From each chosen $s$, we sample $K$ action chunks, roll each out to termination under the current policy, and normalize the binary success rewards within that group as in standard GRPO.

Relation to Prior Work

The mechanism comes from generative-model RL. Flow-GRPO made flow-matching models trainable by GRPO by adding noise to the deterministic sampling path, which provides exploration and tractable action likelihoods. We reuse this directly for action chunks. TempFlow-GRPO showed that branching trajectories within the denoising process improves credit assignment and gives a 10x efficiency gain over Flow-GRPO. It branches inside the image generator's noise schedule. We branch along the physical timeline of an episode, which is a different axis.

Reinforcement Learning for Flow-Matching Policies is the closest GRPO-for-control paper. It samples action groups per state for flow policies and cuts cost 50–85% relative to imitation. However, it works on 2D unicycle dynamics with surrogate rewards and does not address long horizons, where groups collapse.

Four robotics papers motivate or constrain the design. WMPO runs GRPO inside a pixel-space video world model and reaches 57.6% mean success on four MimicGen tasks from 128 real trajectories. Its authors identify two limits: generation cost, and physics errors in contact-rich settings. We keep a real simulator and spend compute on where to branch rather than on dreaming. RedFlow shows that failures are localized at critical actions and fixes them offline. It clusters states with HDBSCAN and applies attraction and repulsion losses, reaching 75.8% on LIBERO-Spatial from 1,536 offline trajectories versus 13,000–24,000 for on-policy baselines. That is the sample-efficiency bar we must clear. Our method finds the critical states by measured outcome variance rather than clustering, and it updates by policy gradient rather than hand-designed losses.

VLA-ATTC applies relative comparison of action candidates at test time only and leaves the policy weights unchanged. We push the relative signal into the weights. RoboCasa reports composite long-horizon success often below 12% even with synthetic demonstration scaling. That is the regime where advantage collapse should be most severe and the proposal most useful.

Research Program

The starting point is a flow-matching VLA fine-tuned by imitation on the standard 50 demonstrations per task. We maintain a branch-point buffer seeded with states sampled uniformly along demonstration trajectories. The buffer is then enriched with states from the policy's own rollouts, especially states shortly before a failure.

Each point keeps a smoothed success estimate updated from every branch launched from it. Points whose estimate saturates near 0 or 1 are down-weighted automatically by the $p(1-p)$ rule. Each training step draws a batch of branch points and samples $K=8$ noisy action chunks per point. It executes each chunk and then continues with the current policy to episode end, which is capped at the remaining task horizon. Finally, it applies the clipped GRPO objective to the branched chunk only, using the within-group normalized advantage. Because every member of a group shares the same state, the advantage isolates the effect of that one decision.

We will also measure what the method changes. Three quantities are tracked. The first is the fraction of groups with nonzero reward spread, which directly measures advantage collapse. The second is where along the episode the selected branch points concentrate. The third is whether those locations coincide with physically meaningful events: grasp onset, contact changes, and subtask boundaries, detected from simulator contact flags.

Baselines

The naive reference is the imitation-only policy trained on the same 50 demonstrations. Every RL method must beat it. It also sets the collapse regime, since we record its per-task initial-state success rate.

The most direct competitor is standard GRPO from initial states. It uses the same flow policy, the same noise-injection scheme from Flow-GRPO, the same $K$, the same binary reward, and the same environment-step budget, with groups formed only at episode start. This isolates the value of branching itself.

The strongest existing method on sample efficiency is RedFlow. It reports 75.8% on LIBERO-Spatial with 1,536 offline trajectories and an overall LIBERO lift from 56.2% to 68.2%. We will run it on the identical base policy and give it the same number of collected trajectories as our method consumes, counting each partial branch rollout as its fraction of a full episode's steps. Beating it means higher success at matched step budget, or equal success at no more than half the steps.

On MimicGen, the strongest group-based competitor is WMPO. It reports 57.6% mean success, against 37.1% for its RL baseline and 42.4% for offline DPO, from 128 trajectories. We report our method at the 128-trajectory budget and also report wall-clock compute, since WMPO's cost is dominated by video generation.

We also include PPO with a learned value network at the same step budget, the natural critic-based answer to collapse. Every method gets the same hyperparameter search budget of 12 configurations, selected on held-out initial states.

Three ablations isolate the key ingredient. The first samples branch points uniformly instead of by $p(1-p)$. The second uses demonstration states only, with no self-generated failure states. The third, the reverse-curriculum control, starts ordinary full GRPO groups from demonstration states without per-point success tracking.

Why not just…?

Why not just generate more synthetic demonstrations and keep imitation learning? RoboCasa shows that 3,000 synthetic demonstrations raise atomic tasks from 28.8% to 47.6%, but composite tasks stay below 12%. Imitation data also contains no recoveries. We will still include a MimicGen-augmented imitation baseline on the same tasks. If it matches our method at equal simulator compute, the RL component is not earning its cost, and we would report that.

Why not just use PPO with a critic, which never collapses? This is a real threat. A value network gives a nonzero learning signal even when all outcomes fail. Our bet is that with 50 demonstrations and sparse binary reward, the critic is itself poorly trained early on and adds variance, which is the instability the GRPO literature was built to avoid. The matched-budget PPO baseline settles this. If PPO equals our method on LIBERO-Long and RoboCasa composites, the claim reduces to "branching helps GRPO" rather than "branching beats critics," and we would scope it that way.

Why not just start ordinary GRPO groups from demonstration states, a plain reverse curriculum? This is the simplest version of the idea and could capture most of the gain. The third ablation is exactly this. The proposal only stands if variance-seeking selection plus self-generated failure states measurably increases the nonzero-spread group fraction and final success over it. If they tie, the contribution shrinks to a clean demonstration that start-state choice cures advantage collapse. That is still useful, but it is a smaller claim.

Why not just do test-time best-of-N selection? VLA-ATTC cuts π0.5's LIBERO-LONG failure rate by over 50% at test time. However, it cannot produce actions the base policy never proposes, and it adds a critic at deployment. We will apply VLA-ATTC-style selection on top of both the imitation policy and our fine-tuned policy. If the gains add, the two methods are complementary. If selection on the imitation policy matches our trained policy, weight updates are unnecessary.

Proposed Experiments

The primary benchmarks are LIBERO-Spatial, -Goal, and -Long from LIBERO, the four MimicGen tasks used by WMPO, and a subset of RoboCasa composite tasks. We report success rate against environment steps, not just final numbers.

The first decisive check is mechanistic. On LIBERO-Long and RoboCasa composites, standard GRPO should show a nonzero-spread group fraction near zero, and branching should keep it substantially higher throughout training. If it does not, the core premise fails.

The second check is efficiency. On LIBERO-Spatial our method should exceed RedFlow's 75.8% using fewer than 1,536 trajectory-equivalents. On MimicGen it should exceed WMPO's reported 57.6% at 128 trajectories with lower compute. For LIBERO-Long we found no absolute success number in the file to anchor against, beyond VLA-ATTC's relative failure-rate reduction, so we report our own re-run baselines.

The third check is robustness. We evaluate on the camera and initial-state perturbation axes of LIBERO-Plus, where OpenVLA drops by over 75% under camera shifts. We test whether training on self-generated off-distribution states improves recovery more than imitation or RedFlow do.

The main limitation is that simulator resets are unavailable on real hardware. A natural follow-up branches from logged real states using a short-horizon learned model. Short branches would keep that model's errors smaller than WMPO's full imagined episodes, but we leave this outside the core claim.