Robot policies trained by imitating a few dozen human demonstrations stall in two predictable ways. On long multi-step tasks they rarely succeed. When they do succeed, they copy the demonstrator's hesitations and can never move faster or more cleanly than the person did. Reinforcement learning can push past both limits. Each standard way of running it, however, needs either a hand-written reward or a learned value network (a "critic" that predicts future reward), and both are fragile.
This note proposes a different training signal for few-demonstration manipulation. The method uses only two learning signals: a single bit from the simulator saying whether the task succeeded, and a learned "realism" score from a discriminator. The discriminator is a classifier trained in a two-player zero-sum game to tell the robot's motions apart from the human demonstrations. Both signals are converted into advantages by group-relative policy optimization (GRPO). GRPO samples several candidate action sequences from the same state and scores each one only relative to its siblings, so no critic is needed.
Core Claim
Inside a GRPO group, the two signals do different jobs at different times. Early in training, most groups contain no successes, so the success bit has zero variance within the group and contributes nothing. In those groups the discriminator's relative ranking is the only gradient, and it pulls the policy toward demonstrator-like motion that eventually succeeds. Once successes appear, the success signal takes over. The discriminator is then demoted from a reward to a constraint: it limits how far the policy may drift from human-like motion, but leaves room to beat the demonstrator on speed.
We claim this combination does two things. It beats success-only GRPO from a behavior-cloned start on tasks where initial success is low. It beats critic-based adversarial imitation at equal simulation budget. And it produces policies that finish faster than the demonstrations without the unnatural, simulator-exploiting motion that pure success optimization tends to find.
We also predict where the claim should fail. On tasks where the behavior-cloned policy already succeeds often, the discriminator should add little. We will test this deliberately.
The per-candidate advantage is
$$A_i = \frac{S_i - \mu_S}{\sigma_S + \epsilon} + \lambda\,\frac{D_i - \mu_D}{\max(\sigma_D, \tau)},$$
where the symbols are as follows:
- $i$ indexes the $G$ candidate action chunks sampled from one simulator state.
- $S_i \in \{0,1\}$ is task success after executing chunk $i$ and continuing with the current policy.
- $D_i$ is the discriminator's mean log-odds that chunk $i$'s transitions came from demonstrations.
- $\mu$ and $\sigma$ are the within-group mean and standard deviation of each signal.
- $\epsilon$ is a small constant, and the success term is zero when $\sigma_S = 0$.
- $\tau$ is a floor on $\sigma_D$.
- $\lambda \ge 0$ is a Lagrange multiplier. It rises whenever the batch-average $D$ falls below a target realism level $-\delta$ and decays otherwise.
Prior Work
The method borrows from three groups of papers and must answer to a fourth.
Reinforcement Learning for Flow-Matching Policies supplies the policy-side recipe. It samples groups of action chunks from a flow-matching policy (a generator that maps noise to action sequences) and updates them with GRPO. It reports 50% to 85% lower cost than imitation-only flow matching. Its gains depend on six hand-written cost terms on planar unicycle dynamics. We keep the critic-free group update but replace those cost terms with one success bit plus an adversarial realism score, and we move to contact-rich manipulation.
Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions and ARC - Actor Residual Critic for Adversarial Imitation Learning supply the reward side. Both show that a discriminator trained against scarce demonstrations can replace hand-engineered style rewards. AMP needs only about 4.5 seconds of dog motion and reaches lower cost of transport than hand-designed rewards. ARC gets more stable updates by differentiating through the discriminator. Both still train a value network: PPO's critic in AMP, a residual critic in ARC. AMP additionally hand-tunes a fixed weight between the style reward and the task reward. Our proposal removes the critic and replaces the fixed weight with an adaptive constraint.
Constrained Group Relative Policy Optimization constrains the design. It shows that adding reward and cost into one number before group normalization lets the higher-variance signal dominate and effectively erases the Lagrange multiplier. It fixes this by normalizing each component separately, which is why our equation normalizes $S$ and $D$ separately. Its costs are fixed indicator functions. Ours is a moving adversary, and how normalization behaves against a moving adversary is untested.
Aligning Multi-Trajectory Supervision with Policy Optimization for VLA Driving gates positive credit on feasibility. That is the reverse of our ordering: we use imitation as the fallback when success gives no signal, not as a gate. Its finding that off-distribution targets inject biased gradients is a warning we take seriously.
On the game-theory side, Last-iterate Convergence of Decentralized Optimistic Gradient Descent/Ascent shows that zero-sum learning dynamics tend to cycle unless damped, for example by optimistic updates. Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Games shows that a KL penalty toward a reference policy makes the game strongly concave, which speeds convergence. GRPO already carries such a penalty toward the behavior-cloned policy.
Your Group-Relative Advantage Is Biased motivates a specific worry. When the discriminator saturates, $\sigma_D$ collapses, and dividing by it inflates noise. The floor $\tau$ exists to prevent this, and we test whether it is needed.
Research Program
Policy and data. The policy is a flow-matching action-chunk model built on the architecture in BAKU: An Efficient Transformer for Multi-Task Policy Learning. BAKU uses a small causal transformer with action chunking and works with about 17 demonstrations per task. We pretrain it by behavior cloning on human demonstrations only, at 50 and 10 demonstrations per task, to sit firmly in the data-scarce regime.
Group sampling. Groups are formed by resetting the MuJoCo-based simulator to a saved state and sampling $G$ chunks through different noise seeds. Each chunk is followed by an on-policy rollout to episode end, which yields its success bit.
Discriminator training. The discriminator scores short windows of proprioception and object state, rather than images, to reduce visual shortcuts. It is trained against demonstration windows with a gradient penalty. We compare plain and optimistic discriminator updates, following the damping result above.
Multiplier schedule. The multiplier $\lambda$ is a dual variable on the constraint that average realism stays above $-\delta$. Sweeping $\delta$ traces a curve from "strict imitation" to "success at any cost." That curve is the direct test of the demonstrator-ceiling question: how much faster than the human the policy can become before its motion stops looking human.
Analysis of mechanism. We log three quantities over training. The first is the fraction of groups with zero success variance, which should fall as the discriminator does its job. The second is the trajectory of $\sigma_D$. The third is how often the floor $\tau$ is active.
Baselines
The naive reference point is the behavior-cloned flow-matching policy itself, trained on the same 50 or 10 human demonstrations and evaluated with no further training. We also run a version trained on MimicGen-expanded data, following RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots. RoboCasa reports that augmenting 50 human demonstrations with 3,000 generated ones raised atomic-task success from 28.8% to 47.6%, and that composite long-horizon tasks often remain below 12% success. These are the reported numbers our method must clear on those task families.
The strongest competitor on this specific question is success-only GRPO from the same behavior-cloned start. This is the Pfrommer et al. recipe with the six cost terms replaced by the success bit. It uses the same simulator resets, group size, KL coefficient, and environment-step budget. If it matches us, the discriminator is pointless. No reported result was found on Lacuna for GRPO on RoboCasa or LIBERO manipulation, so we produce this number ourselves. Our method beats it if, on tasks with initial success below 15%, it reaches higher final success at equal environment steps, judged by non-overlapping 95% confidence intervals over five seeds.
The critic-based adversarial competitor is AMP-style PPO. It uses the same discriminator architecture and the same success bit, with the hand-tuned fixed weight replaced by our dual-variable schedule so that the only difference is the critic. No reported result was found on Lacuna for AMP or ARC on RoboCasa or LIBERO.
Three ablations isolate the key ingredients:
- Normalization. Naive scalarization (normalizing $S + \lambda D$ as one number) against our component-wise normalization.
- Adversary. A discriminator frozen after pretraining against one that keeps updating, which tests whether the zero-sum game matters at all.
- Variance floor. Standard-deviation scaling with and without the floor $\tau$.
Fairness rules. Every method gets the same demonstration set, the same simulation budget, and the same hyperparameter search budget of twelve configurations per method. Checkpoints are chosen on held-out scene layouts, never on test scenes.
Why not just…?
Why not just run success-only GRPO from a good behavior-cloned policy? This is the real threat. On atomic RoboCasa tasks the behavior-cloned policy may already succeed often enough that groups carry success variance, and then the discriminator should add nothing. We predict exactly this and will report it. Our claim rests on two other regimes. The first is composite tasks, where RoboCasa reports success often below 12%, so most groups are all-failure and give zero gradient. The second is motion quality: success-only optimization has no reason to avoid jerky or simulator-exploiting motion. If success-only GRPO matches us on composite tasks and on smoothness, the idea is wrong.
Why not just keep the critic, as AMP and ARC do? Critic-based adversarial imitation clearly works. ARC reports about 6306 return on Ant-v2 against about 5949 for f-MAX. The question is whether a critic chasing a moving discriminator reward is a liability at small data and long horizons. Wei et al. needed a deliberately slow critic just to stop oscillation. The matched AMP-PPO comparison settles this. If the critic-based version wins at equal steps, the contribution shrinks to a negative result about critic-free adversarial training, and we would report it that way.
Why not just add a KL penalty toward the behavior-cloned policy instead of a discriminator? A KL penalty keeps the policy near its own imperfect initialization, not near the demonstration data. It also cannot reward unfamiliar states that look like the demonstrations. The frozen-discriminator ablation and a KL-only variant with matched strength separate "stay close to the start" from "look like the data." If KL-only matches us, the adversarial game is unnecessary.
Why not just use a vision-language model to score trajectories? RoboVQA is a benchmark of question answering about robot videos. On it, even the specialized ACE-Brain-0 reaches only 64.6%, and CoRe documents hallucination and reward gaming when such models are used as reward sources. Fine contact timing is exactly where these models are weakest, so we treat this option as out of scope rather than as a competitor.
Decisive Experiments
Benchmarks. The main testbed is RoboCasa. We use atomic and composite tasks with 50 and 10 human demonstrations, and evaluate on held-out kitchen layouts. The secondary testbed is the long-horizon portion of LIBERO. On LIBERO-90, BAKU already reports 90% success, so we use LIBERO only with 10 demonstrations per task, where headroom exists.
Metrics. We measure success rate, episode length relative to the mean demonstration length, action jerk, and discriminator realism on held-out demonstrations. We also run a hacking audit: a manual review of fifty successful rollouts per method for physically implausible contacts.
Decisive check 1: a crossover with initial success. The gain over success-only GRPO should shrink as the behavior-cloned policy's initial success rises. It should be large on composite tasks and near zero on atomic tasks with initial success above 40%. The fraction of zero-variance groups should fall faster under our method. A flat gain across difficulty would mean the mechanism is not what we claim.
Decisive check 2: beating the demonstrator without losing realism. As $\delta$ is relaxed, episode length should drop below the demonstrator mean, while jerk and audit failures stay at behavior-cloning levels. Success-only GRPO should reach similar speed only with higher jerk and more audit failures.
Decisive check 3: stability of the critic-free game. Without the floor $\tau$, or under naive scalarization, we expect oscillating realism scores and late-training collapse. With component-wise normalization and the floor, the last iterate should be stable. If both variants are equally stable, the normalization concern raised by Constrained GRPO does not carry over to adversarial signals, and we would report that as a finding in its own right.
References
- Reinforcement Learning for Flow-Matching Policies
- Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions
- ARC - Actor Residual Critic for Adversarial Imitation Learning
- Constrained Group Relative Policy Optimization
- Aligning Multi-Trajectory Supervision with Policy Optimization for VLA Driving
- Last-iterate Convergence of Decentralized Optimistic Gradient Descent/Ascent in Infinite-horizon Competitive Markov Games
- Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Games
- Your Group-Relative Advantage Is Biased
- BAKU: An Efficient Transformer for Multi-Task Policy Learning
- RoboCasa: Large-Scale Simulation of Household Tasks for Generalist Robots
- ACE-Brain-0: Spatial Intelligence as a Shared Scaffold for Universal Embodiments
- CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning