TL;DR
This paper introduces a generative data augmentation framework that leverages pre-trained text-to-image diffusion models to bootstrap small robot datasets into diverse, semantically rich training data. By automatically altering object textures, backgrounds, and distractors while keeping the required actions identical, the method significantly improves the real-world generalization of robot manipulation policies at zero additional human demonstration cost.
What Problem Does This Paper Address?
For robotic manipulation policies to work reliably in the real world, they must generalize to unseen environments, novel objects, and cluttered backgrounds. Achieving this level of robustness typically requires massive, diverse datasets. However, collecting real-world robot demonstrations via teleoperation is incredibly expensive, time-consuming, and logistically rigid.
Traditional data augmentation techniques in machine learning—such as cropping, color jittering, and Gaussian blur—help with minor perceptual variations but fail to introduce structured, semantic changes. They cannot simulate a completely new kitchen countertop, add realistic distractor objects, or change a basket into a plastic bucket. This paper addresses this bottleneck by using the rich world priors embedded in large, pre-trained image-text generative models to synthesize these complex variations automatically.
Methods
The authors propose a Generative Augmentation framework that applies text-to-image models to enforce semantic invariance—ensuring that varied but semantically related visual observations map to the exact same robotic action.
The framework operates in two distinct regimes depending on data availability and the need for structural control:
- Structure-Aware Augmentation (Low-Data Regime): Designed for settings with limited initial data where physical plausibility is critical. It utilizes 3D object meshes and manual segmentation masks combined with depth-aware diffusion models. This allows the system to perform both in-category changes (e.g., swapping textures while preserving 3D geometry) and cross-category changes (e.g., replacing an entire object category and rendering it realistically into the scene using the original camera pose).
- Scalable Augmentation (Multi-Task Video Regime): A fully automated pipeline tailored for large video datasets. It uses the robot's joint angles and forward kinematics to locate the end-effector, prompts SegmentAnything (SAM) to mask the manipulated object, and tracks it across video frames using TrackAnything. Pre-trained inpainting models then automatically alter the object or background based on randomized text prompts.
For policy learning, the augmented data is fed into language-conditioned architectures: CLIPort for single-step pick-and-place tasks, and a multi-task Transformer-based Conditional Variational Autoencoder (CVAE) policy for complex, multi-step kitchen tasks.
Datasets
The framework was evaluated across both simulated and real-world benchmarks:
- Real-World Pick-and-Place: A dataset of 10 tabletop tasks with only 10 demonstrations per task (100 total), initially collected in a single, simple environment. Policies were evaluated on entirely unseen, highly cluttered test environments ("Playground", "Study Desk", "Kitchen Island", "Garage", "Bathroom").
- Real-World Multi-Task Kitchen: A large-scale dataset consisting of 7,500 teleoperated trajectories spanning 38 distinct kitchen manipulation tasks (e.g., opening drawers, pouring, pushing objects) collected over two months.
- Simulation Benchmarks: Tabletop pick-and-place tasks utilizing 40 object meshes from the GoogleScan and Free3D datasets, alongside a multi-task kitchen environment featuring 18 tasks across 100 randomized layout variations.
Key Insights and Ideas
- Web-Scale Priors as Robot Data Sources: Instead of manually modeling environments or collecting endless physical data, researchers can treat pre-trained diffusion models as a repository of "world common sense" to inject realistic visual diversity into training.
- Preserving the Action Space: The core insight is that visual and semantic context can be heavily altered without changing the underlying physics of the trajectory. A pick-and-place motion remains valid whether the table is made of wood or marble.
- Geometry Matters for Robotics: Naive 2D image inpainting often violates physical constraints (e.g., floating objects or clipping). Integrating depth maps and 3D assets ensures that the augmented data remains geometrically consistent, which is vital for spatial reasoning models like CLIPort.
Key Results
- Pick-and-Place Generalization: In real-world testing, the generative augmentation approach (GenAug) achieved an average 85% success rate in entirely unseen environments, compared to just 38% for the baseline trained without augmentations.
- Unseen Objects: When tested on completely novel objects, GenAug achieved a 52% success rate for placing (vs. 8% baseline) and 45% for picking (vs. 10% baseline).
- Multi-Task Kitchen Scaling: For complex kitchen tasks, the proposed method (MT-AUG) significantly outperformed standard visual imitation learning and competitive baselines (like RT-1 and CACTI). It demonstrated a 400% relative improvement in the highest level of generalization (L3), which involves entirely new combinations of skills and objects.
Limitations and Future Work
- The "Action Assumption": The framework assumes that changing an object's appearance does not change how the robot should physically interact with it. However, changing an object's material (e.g., from plastic to glass) alters real-world physics parameters like friction, mass, and compliance, which this visual-only augmentation method cannot account for.
- Computational Speed: Generating these augmentations takes roughly 30 seconds per scene. While perfectly fine for offline behavioral cloning, it is currently too slow to be used in-the-loop for online reinforcement learning.
- Visual-Only Focus: The framework does not augment actions or compose entirely new long-horizon tasks. The authors note that leveraging video generation models to augment data at the action level is a promising future direction.
Summary updated