Robot imitation learning, where a policy is trained to copy recorded human demonstrations, stops improving once the demonstrations stop covering the situations the robot meets. Off the robot, the usual fix is to generate more data. A simulator re-plays transformed demonstrations and throws away the failures, or a large model generates samples that a second model then filters. Both are expensive. On a real robot without a simulator there is often no cheap way at all to check whether a synthetic demonstration is valid. We propose a generator that produces a camera image and its pixel-wise object labelling (a segmentation mask) together, conditioned on a requested object layout. The mask the generator draws then serves as a nearly free self-check: if it disagrees with the layout we asked for, or with what the action implies (a closed gripper that touches nothing), the sample is discarded. Because a diffusion model, a generator that turns noise into a sample over many small denoising steps, commits to coarse geometry early, this check can run partway through generation, so rejected samples cost only a fraction of a full sample.
The claim to test has two parts. First, agreement between the generated mask and the requested layout predicts whether a synthetic frame will help or hurt a downstream policy, and it does so about as well as simulator replay, which is the expensive check it would replace. Second, filtering on this signal lets a few-shot generator, trained on about ten demonstrations, add data past the point where unfiltered synthetic data stops helping. We do not claim to break the imitation ceiling caused by poor demonstrator skill. We target the part of the ceiling caused by poor coverage, and we design the evaluation to keep these two parts separate.
Where This Sits
Joint image–mask generation exists only in static vision. One-Shot Synthesis of Images and Segmentation Masks (OSMIS) trains a GAN with a two-branch generator from a single labelled image, and it reports a DAVIS-17 mask-alignment mIoU of 86.6. It cannot change object pose, and it has no notion of actions. Addressing Data Scarcity in Materials Science Research with Deep Generative Models concatenates images with masks inside a diffusion model trained on 200 frames, and Synthetic Augmentation for Anatomical Landmark Localization using DDPMs does the same with landmark heatmaps. The materials paper matters for our design because it is a negative result: gains plateau or reverse beyond about 5,000 synthetic pairs, and no remedy is offered. We take the concatenated-channel diffusion design from these papers. We change its use in two ways: the mask becomes a verifier rather than only a training label, and the output becomes action-labelled demonstration frames.
In robotics, MimicGen supplies the action side of our method. It splits about 200 human demonstrations into segments defined relative to objects, transforms those segments to new object poses, and grows them into more than 50,000 trajectories. It keeps only trajectories that succeed when re-executed in simulation, so it needs CAD assets and many simulator rollouts. RoCoDA applies rigid transforms to object poses with matching corrections to the actions, swaps task-irrelevant parts of the scene, and adds colour jitter. It reports 99.3% versus 22.3% for a vanilla policy on Libero-Object with unseen distractors, and 15.7% versus MimicGen's 9.3% on Three-Piece Assembly. It needs known object states and cannot render new visual geometry. Predicting What Matters (the Mask World Model, MWM) shows that predicting future masks instead of future pixels makes policies much more robust: 27.5% versus 3.8% for GE-ACT under LIBERO background shift. It consumes masks but does not create new demonstrations.
Our method borrows the analytic action transformation used by MimicGen and RoCoDA. It replaces simulator rendering and replay with joint image–mask generation followed by a mask self-check. Robustness methods like MWM are not what we replace. They are a competitor we must beat or combine with.
Two lines of work constrain the claims we can make. Large Decision Models, Diffusion Policy and Beyond Imitation argue that supervised imitation reproduces the demonstrator's mistakes. Offline Q-learning on Diverse Multi-Task Data shows that stitching good pieces of suboptimal trajectories together with a learned value function can exceed dataset quality. Its Atari agents reach 77.8% human-level performance from data that averages 51%. Our synthetic data reuses the demonstrator's own action segments, so it cannot beat the demonstrator's skill. That is the reason we limit the claim to coverage.
On the overhead side, DISCO and TrueTeacher show the same pattern in language data: a costly generation step followed by a second, costly verification step. Beyond Quantity finds that adding synthetic data without regard to diversity saturates early. Our verifier comes out of the generator itself, so it does not need a second model.
Core Mechanism
We start from a few real demonstrations and their masks; masks for real frames can come from an off-the-shelf segmenter. We sample a new object configuration and transform the object-relative action segments, following MimicGen. Rendering the object masks at their new poses gives a requested layout mask $L$. A diffusion model is trained on concatenated RGB and mask channels, conditioned on $L$ and on the robot's arm configuration. It denoises toward a joint sample. At an intermediate step $k$ of $T$, the model's current estimate of the clean output includes a predicted mask $\hat{M}_k$. We score the sample as
$$s_k = \mathrm{IoU}(\hat{M}_k, L) \cdot c(\hat{M}_k, g),$$
where $\mathrm{IoU}$ is the overlap between the predicted and requested object regions, averaged over objects. The second factor, $c$, is a contact-consistency term computed from the action labels: it is 1 if the predicted gripper region overlaps the target object exactly when the gripper command $g$ is "closed" and is in a contact phase, and it decays toward 0 otherwise. Samples with $s_k$ below a threshold $\tau$ stop generating at step $k$. Survivors finish denoising and are scored again on their final mask.
The policy itself is unchanged. It is a standard Diffusion Policy or an Action Chunking Transformer (ACT), trained on the real frames plus the retained synthetic frames.
Because the generator itself is trained on very few demonstrations, it faces the few-shot overfitting problem: it may memorise and copy training frames. A high mask-agreement score does not detect copying. We therefore also report each sample's perceptual distance (LPIPS) to its nearest training frame. In one ablation we add a diversity floor alongside the threshold $\tau$, so that we can tell whether the late-stage saturation is caused by invalid samples or by redundant ones.
Research Program
We first validate the verifier in simulation, where the truth is available. We use LIBERO and robosuite tasks, generate candidate frames with their transformed actions, and label each candidate with the expensive check: we re-execute the transformed segment in the simulator and record whether the rendered object layout matches. We then measure how well $s_k$ predicts that label (precision, recall and AUROC) at $k/T \in \{0.1, 0.25, 0.5, 1.0\}$. This tells us how early rejection can safely happen.
Second, we measure overhead as GPU-seconds per retained sample and, more importantly, policy success per GPU-hour of data generation. Simulator replay, post-hoc SAM checking and our early check are compared at a matched number of retained samples.
Third, we sweep the number of synthetic samples from 500 to 20,000, with and without filtering. The question is whether filtering moves or removes the saturation point that the materials-science paper reports at about 5,000.
Fourth, we run a ceiling decomposition. Each method is evaluated on two sets of initial states: those within the support of the demonstrations, and those deliberately outside it. We repeat this with deliberately suboptimal demonstrations, such as hesitant or partly failed grasps. If our method fixes coverage, gains should appear mainly on the out-of-support states and should not repair the suboptimal-demonstration condition. A gain on suboptimal demonstrations would point to something other than coverage and would need explaining.
Baselines
The do-nothing reference is the same Diffusion Policy or ACT trained only on the real few-shot demonstrations with standard crop and colour-jitter augmentation. The file gives a vanilla figure only for Libero-Object with unseen distractors: 22.3% success, reported in RoCoDA.
The strongest augmentation competitor is RoCoDA itself, at 99.3% on Libero-Object with distractors and 15.7% on Three-Piece Assembly. Because it is near the ceiling on distractors, we do not expect to beat it there. Our comparison targets the settings it cannot handle: background shift, and real scenes without object-state access. We run it with the same demonstrations and the same policy architecture. Both methods get the same number of retained synthetic samples and the same hyperparameter search budget of 20 configurations. Beating it means higher success under LIBERO background shift, or on real-robot tasks, with no loss on its in-distribution tasks.
The strongest robustness competitor is MWM, which reports 98.3% average on LIBERO and 27.5% under background shift. We use its published numbers where our protocol matches; otherwise we re-train it on our demonstration counts. Beating it means exceeding 27.5% under background shift. We also test the combination of MWM trained on our synthetic data, because the two approaches may be complementary rather than rival.
MimicGen with simulator replay is the oracle filter in simulation. It gets the same generator samples and the same retention rate. Matching its downstream success at a small fraction of its generation cost counts as success.
Two ablations isolate the key ingredient. The first is the same joint generator with no filtering, which is the naive scaling curve. The second is a random filter with the same retention rate, which separates the effect of keeping fewer samples from the effect of keeping the right ones. A third ablation replaces the internal mask with a SAM segmentation of the finished RGB image, which isolates whether the joint mask channel matters at all.
Why not just…?
Why not just generate RGB with an off-the-shelf image editor and label it with SAM afterwards? This is the most serious threat, and it is a direct ablation in our program. The SAM-based check needs a finished image, so it cannot reject a sample early, and it adds a second large model's inference to every candidate. However, if it predicts simulator validity much better than $s_k$ does, then our main argument rests only on cost savings. We will report both AUROC and GPU-seconds per retained sample so that a reader can judge the trade-off. If SAM matches our verifier in accuracy and the cost gap is small, the joint-generation part of our claim fails.
Why not just use procedural augmentation like RoCoDA? On tasks where object states are known and the visual shift consists of distractors, RoCoDA's 99.3% leaves little room, and we concede that regime. Its augmentations are analytic swaps and colour jitter, so it cannot produce a new background or new appearance at a new pose. MWM's numbers show that background shift breaks even mask-aware policies, at 27.5%. The background-shift and real-robot experiments decide whether generation adds anything beyond RoCoDA.
Why not just freeze a strong pretrained embedding? Rethinking Few-Shot Image Classification shows that freezing a pretrained backbone and fitting a linear classifier beats meta-learning, reaching 82.14% on miniImageNet 5-way 5-shot. The same paper notes that this approach degrades under cross-domain shift. A frozen embedding also cannot add training states the demonstrations never visited. We include a frozen-encoder policy baseline. If it matches our method on the out-of-support initial states, the coverage argument is wrong.
Why not just keep generating unfiltered data, or use a learned perturbation like Meta Dropout? The materials study reports gains reversing beyond about 5,000 synthetic pairs, and our scaling sweep tests this directly. Meta Dropout learns input-dependent noise on internal features and lifts MAML on miniImageNet 1-shot from 49.58% to 51.93%. It requires second-order gradients, and it perturbs features rather than adding states the robot has never seen. For these reasons we treat it as a secondary comparison and not as the main threat.
Proposed Experiments and Decisive Checks
Simulation experiments use LIBERO (the full suite and the background-shift stress test), Libero-Object, and robosuite's Three-Piece Assembly and Coffee tasks. We train with 5, 10 and 25 demonstrations per task, three seeds each, and run at least 50 evaluation rollouts per task and seed. Real-robot experiments use a Franka arm on pick-and-place and drawer opening, with no simulator. For latency of the deployed policy we optionally use STEP-style warm starting (STEP); this affects only deployment cost and is not part of the claim.
Three results decide the claim. The first is whether $s_k$ at $k/T \le 0.25$ predicts the simulator-replay label with an AUROC that clearly separates it from the SAM-based check at lower cost. If it does not, the overhead claim fails. The second is whether filtered synthetic data keeps improving success beyond the size where the unfiltered curve flattens or falls, while the random filter at the same retention rate does not. If both filters give the same curve, the gain comes from keeping fewer samples, not from verification. The third is whether the gains over RoCoDA and MWM appear on out-of-support and background-shift states, and do not appear under suboptimal demonstrations. That pattern would confirm the coverage interpretation, and any other pattern would refute it.
References
- One-Shot Synthesis of Images and Segmentation Masks
- Addressing Data Scarcity in Materials Science Research with Deep Generative Models
- Synthetic Augmentation for Anatomical Landmark Localization Using DDPMs
- MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations
- RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrations
- Predicting What Matters: Robust Generalist Robot Policy Learning via Future Semantic Mask
- Large Decision Models
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
- Offline Q-learning on Diverse Multi-Task Data Both Scales And Generalizes
- DISCO: Distilling Counterfactuals with Large Language Models
- TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
- Beyond Quantity: Trajectory Diversity Scaling for Code Agents
- Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?
- Meta Dropout: Learning to Perturb Latent Features for Generalization
- STEP: Warm-Started Visuomotor Policies with Spatiotemporal Consistency Prediction