Abstract
Robotic manipulation policies often fail when encountering visual distractors not present during training. We present NICE (Naturalistic Inpainting for Context Enhancement), a framework that diversifies real-world demonstration data through automated "scene surgery." By decomposing scenes and applying targeted edits—removal, restyling, and replacement of distractors—NICE generates diverse training scenarios that maintain action-label consistency. Experimental results demonstrate significant improvements in affordance prediction and manipulation success rates in cluttered environments without additional physical data collection.
Introduction
The "brittleness" of robotic systems remains a significant hurdle for real-world deployment. A policy trained in a controlled laboratory environment often fails when faced with out-of-distribution (OOD) visual distractors, such as everyday objects left in the robot's workspace. These distractors create a gap between training and deployment. Traditional solutions, like massive data collection or simulation, are either prohibitively expensive or suffer from the sim-to-real gap. NICE addresses this by leveraging generative AI to augment existing real-world datasets, creating robust policies through naturalistic scene manipulation.
The NICE Framework
The NICE pipeline treats data augmentation as a multi-stage "scene surgery" process that transforms standard demonstration data into a diverse set of training scenarios.
1. Scene Decomposition
The system first identifies all objects within a frame. This is achieved using Florence-2 for object detection and SAM-2 (Segment Anything Model) for precise segmentation masking. This step allows the system to isolate individual elements for manipulation.
2. Role Assignment
Based on the task instruction (e.g., "Pick up the apple"), the system distinguishes between the target object and distractor objects. This ensures that the core task remains intact while the surrounding context is varied.
3. Scene Editing
Three primary surgical operations are performed on distractors:
- Removal: Objects are erased using the LaMa inpainting model, which reconstructs the background and shadows to maintain visual realism.
- Restyling: Swaps textures or colors using the Describable Textures Dataset while keeping the object's original shape and position.
- Replacement: An LLM (Deepseek-r1) suggests semantically plausible replacements (e.g., replacing a spoon with a whisk), which are then inpainted using Stable Diffusion.
Crucially, these edits are constrained to ensure action-label consistency; the target object's position and the robot's recorded movements remain unchanged, ensuring the augmented data remains valid for training.
Experimental Results
The framework was validated using the Bridge and RoboPoint datasets, alongside 864 real-world trials. Key findings include:
- Affordance Prediction: Accuracy in identifying where a robot can interact with objects in cluttered scenes improved by over 20%.
- Success Rates: Manipulation tasks (picking, placing, stacking) saw an 11% average increase in success rates within heavy clutter.
- Precision and Safety: Target confusion was reduced by 6%, and collision rates with distractors dropped by 7%.
- Efficiency: Synthetic clutter generated by NICE proved as effective for training as manually collected real-world clutter, significantly reducing the need for physical data collection.
Discussion and Limitations
While NICE significantly enhances robustness, it currently focuses on static image edits. Future iterations could address dynamic consistency (how distractors react to robot contact) and explicit 3D geometric modeling to improve perspective accuracy in complex scenes. Expanding the framework to long-horizon tasks remains a promising direction for future research.
Conclusion
NICE demonstrates that generative AI can serve as a powerful "data engine" for robotics. By performing surgical, naturalistic edits on real-world data, it bridges the gap between lab-trained policies and the messy reality of human environments, providing a scalable path toward more robust robotic manipulation.
Summary updated