Semantically Controllable Augmentations for Generalizable Robot Learning

Zoey Chen, Zhao Mandi, Homanga Bharadhwaj, Mohit Sharma, Shuran Song, Abhishek Gupta, Vikash Kumar

TL;DR

This paper introduces a generative data augmentation framework that leverages pre-trained text-to-image diffusion models to bootstrap small robot datasets into diverse, semantically rich training data. By automatically altering object textures, backgrounds, and distractors while keeping the required actions identical, the method significantly improves the real-world generalization of robot manipulation policies at zero additional human demonstration cost.

What Problem Does This Paper Address?

For robotic manipulation policies to work reliably in the real world, they must generalize to unseen environments, novel objects, and cluttered backgrounds. Achieving this level of robustness typically requires massive, diverse datasets. However, collecting real-world robot demonstrations via teleoperation is incredibly expensive, time-consuming, and logistically rigid.

Traditional data augmentation techniques in machine learning—such as cropping, color jittering, and Gaussian blur—help with minor perceptual variations but fail to introduce structured, semantic changes. They cannot simulate a completely new kitchen countertop, add realistic distractor objects, or change a basket into a plastic bucket. This paper addresses this bottleneck by using the rich world priors embedded in large, pre-trained image-text generative models to synthesize these complex variations automatically.

Methods

The authors propose a Generative Augmentation framework that applies text-to-image models to enforce semantic invariance—ensuring that varied but semantically related visual observations map to the exact same robotic action.

Figure 1. Our Framework takes a small offline dataset containing expert demonstrations and leverages text-to-image generative models to semantically bootstrap the initial dataset into a much larger and diverse augmented dataset, which can be used to train a robot policy that generalizes to unseen environments and tasks.

The framework operates in two distinct regimes depending on data availability and the need for structural control:

  1. Structure-Aware Augmentation (Low-Data Regime): Designed for settings with limited initial data where physical plausibility is critical. It utilizes 3D object meshes and manual segmentation masks combined with depth-aware diffusion models. This allows the system to perform both in-category changes (e.g., swapping textures while preserving 3D geometry) and cross-category changes (e.g., replacing an entire object category and rendering it realistically into the scene using the original camera pose).
  2. Scalable Augmentation (Multi-Task Video Regime): A fully automated pipeline tailored for large video datasets. It uses the robot's joint angles and forward kinematics to locate the end-effector, prompts SegmentAnything (SAM) to mask the manipulated object, and tracks it across video frames using TrackAnything. Pre-trained inpainting models then automatically alter the object or background based on randomized text prompts.
Figure 2. Our framework provides the ability to augment the scene by changing the object texture (first row), changing the background (second row), adding distractors (third row) and changing object categories (fourth row)

For policy learning, the augmented data is fed into language-conditioned architectures: CLIPort for single-step pick-and-place tasks, and a multi-task Transformer-based Conditional Variational Autoencoder (CVAE) policy for complex, multi-step kitchen tasks.

Datasets

The framework was evaluated across both simulated and real-world benchmarks:

  • Real-World Pick-and-Place: A dataset of 10 tabletop tasks with only 10 demonstrations per task (100 total), initially collected in a single, simple environment. Policies were evaluated on entirely unseen, highly cluttered test environments ("Playground", "Study Desk", "Kitchen Island", "Garage", "Bathroom").
  • Real-World Multi-Task Kitchen: A large-scale dataset consisting of 7,500 teleoperated trajectories spanning 38 distinct kitchen manipulation tasks (e.g., opening drawers, pouring, pushing objects) collected over two months.
  • Simulation Benchmarks: Tabletop pick-and-place tasks utilizing 40 object meshes from the GoogleScan and Free3D datasets, alongside a multi-task kitchen environment featuring 18 tasks across 100 randomized layout variations.

Key Insights and Ideas

  • Web-Scale Priors as Robot Data Sources: Instead of manually modeling environments or collecting endless physical data, researchers can treat pre-trained diffusion models as a repository of "world common sense" to inject realistic visual diversity into training.
  • Preserving the Action Space: The core insight is that visual and semantic context can be heavily altered without changing the underlying physics of the trajectory. A pick-and-place motion remains valid whether the table is made of wood or marble.
  • Geometry Matters for Robotics: Naive 2D image inpainting often violates physical constraints (e.g., floating objects or clipping). Integrating depth maps and 3D assets ensures that the augmented data remains geometrically consistent, which is vital for spatial reasoning models like CLIPort.

Key Results

  • Pick-and-Place Generalization: In real-world testing, the generative augmentation approach (GenAug) achieved an average 85% success rate in entirely unseen environments, compared to just 38% for the baseline trained without augmentations.
  • Unseen Objects: When tested on completely novel objects, GenAug achieved a 52% success rate for placing (vs. 8% baseline) and 45% for picking (vs. 10% baseline).
  • Multi-Task Kitchen Scaling: For complex kitchen tasks, the proposed method (MT-AUG) significantly outperformed standard visual imitation learning and competitive baselines (like RT-1 and CACTI). It demonstrated a 400% relative improvement in the highest level of generalization (L3), which involves entirely new combinations of skills and objects.

Limitations and Future Work

  • The "Action Assumption": The framework assumes that changing an object's appearance does not change how the robot should physically interact with it. However, changing an object's material (e.g., from plastic to glass) alters real-world physics parameters like friction, mass, and compliance, which this visual-only augmentation method cannot account for.
  • Computational Speed: Generating these augmentations takes roughly 30 seconds per scene. While perfectly fine for offline behavioral cloning, it is currently too slow to be used in-the-loop for online reinforcement learning.
  • Visual-Only Focus: The framework does not augment actions or compose entirely new long-horizon tasks. The authors note that leveraging video generation models to augment data at the action level is a promising future direction.

Summary updated

Abstract

Generalization to unseen real-world scenarios for robot manipulation requires exposure to diverse datasets during training. However, collecting large real-world datasets is intractable due to high operational costs. For robot learning to generalize despite these challenges, it is essential to leverage sources of data or priors beyond the robot's direct experience. In this work, we posit that image-text generative models, which are pre-trained on large corpora of web-scraped data, can serve as such a data source. These generative models encompass a broad range of real-world scenarios beyond a robot's direct experience and can synthesize novel synthetic experiences that expose robotic agents to additional world priors aiding real-world generalization at no extra cost. In particular, our approach leverages pre-trained generative models as an effective tool for data augmentation. We propose a generative augmentation framework for semantically controllable augmentations and rapidly multiplying robot datasets while inducing rich variations that enable real-world generalization. Based on diverse augmentations of robot data, we show how scalable robot manipulation policies can be trained and deployed both in simulation and in unseen real-world environments such as kitchens and table-tops. By demonstrating the effectiveness of image-text generative models in diverse real-world robotic applications, our generative augmentation framework provides a scalable and efficient path for boosting generalization in robot learning at no extra human cost.
BibTeX
@article{chen2025semantically,
  author = {Zoey Chen and Zhao Mandi and Homanga Bharadhwaj and Mohit Sharma and Shuran Song and Abhishek Gupta and Vikash Kumar},
  title = {{Semantically Controllable Augmentations for Generalizable Robot Learning}},
  journal = {Int. J. Robotics Res.},
  year = {2025},
  doi = {10.1177/02783649241273686},
}
Figure