Improving Robotic Manipulation Robustness via NICE Scene Surgery

Sajjad Pakdamansavoji, Mozhgan Pourkeshavarz, Adam Sigal, Zhiyuan Li, Rui Heng Yang, Amir RasouliarXiv 2025 · DOI · arXiv

Abstract

Robotic manipulation policies often fail when encountering visual distractors not present during training. We present NICE (Naturalistic Inpainting for Context Enhancement), a framework that diversifies real-world demonstration data through automated "scene surgery." By decomposing scenes and applying targeted edits—removal, restyling, and replacement of distractors—NICE generates diverse training scenarios that maintain action-label consistency. Experimental results demonstrate significant improvements in affordance prediction and manipulation success rates in cluttered environments without additional physical data collection.

Introduction

The "brittleness" of robotic systems remains a significant hurdle for real-world deployment. A policy trained in a controlled laboratory environment often fails when faced with out-of-distribution (OOD) visual distractors, such as everyday objects left in the robot's workspace. These distractors create a gap between training and deployment. Traditional solutions, like massive data collection or simulation, are either prohibitively expensive or suffer from the sim-to-real gap. NICE addresses this by leveraging generative AI to augment existing real-world datasets, creating robust policies through naturalistic scene manipulation.

The NICE Framework

The NICE pipeline treats data augmentation as a multi-stage "scene surgery" process that transforms standard demonstration data into a diverse set of training scenarios.

The NICE framework pipeline: from robot demo data to enhanced datasets via automated scene surgery.

1. Scene Decomposition

The system first identifies all objects within a frame. This is achieved using Florence-2 for object detection and SAM-2 (Segment Anything Model) for precise segmentation masking. This step allows the system to isolate individual elements for manipulation.

Scene decomposition using vision-language models to detect and mask objects for targeted editing.

2. Role Assignment

Based on the task instruction (e.g., "Pick up the apple"), the system distinguishes between the target object and distractor objects. This ensures that the core task remains intact while the surrounding context is varied.

3. Scene Editing

Three primary surgical operations are performed on distractors:

  • Removal: Objects are erased using the LaMa inpainting model, which reconstructs the background and shadows to maintain visual realism.
  • Restyling: Swaps textures or colors using the Describable Textures Dataset while keeping the object's original shape and position.
  • Replacement: An LLM (Deepseek-r1) suggests semantically plausible replacements (e.g., replacing a spoon with a whisk), which are then inpainted using Stable Diffusion.

Crucially, these edits are constrained to ensure action-label consistency; the target object's position and the robot's recorded movements remain unchanged, ensuring the augmented data remains valid for training.

Experimental Results

The framework was validated using the Bridge and RoboPoint datasets, alongside 864 real-world trials. Key findings include:

  • Affordance Prediction: Accuracy in identifying where a robot can interact with objects in cluttered scenes improved by over 20%.
  • Success Rates: Manipulation tasks (picking, placing, stacking) saw an 11% average increase in success rates within heavy clutter.
  • Precision and Safety: Target confusion was reduced by 6%, and collision rates with distractors dropped by 7%.
  • Efficiency: Synthetic clutter generated by NICE proved as effective for training as manually collected real-world clutter, significantly reducing the need for physical data collection.

Discussion and Limitations

While NICE significantly enhances robustness, it currently focuses on static image edits. Future iterations could address dynamic consistency (how distractors react to robot contact) and explicit 3D geometric modeling to improve perspective accuracy in complex scenes. Expanding the framework to long-horizon tasks remains a promising direction for future research.

Conclusion

NICE demonstrates that generative AI can serve as a powerful "data engine" for robotics. By performing surgical, naturalistic edits on real-world data, it bridges the gap between lab-trained policies and the messy reality of human environments, providing a scalable path toward more robust robotic manipulation.

Summary updated

Abstract

Learning robust visuomotor policies for robotic manipulation remains a challenge in real-world settings, where visual distractors can significantly degrade performance and safety. In this work, we propose an effective and scalable framework, Naturalistic Inpainting for Context Enhancement (NICE). Our method minimizes out-of-distribution (OOD) gap in imitation learning by increasing visual diversity through construction of new experiences using existing demonstrations. By utilizing image generative frameworks and large language models, NICE performs three editing operations, object replacement, restyling, and removal of distracting (non-target) objects. These changes preserve spatial relationships without obstructing target objects and maintain action-label consistency. Unlike previous approaches, NICE requires no additional robot data collection, simulator access, or custom model training, making it readily applicable to existing robotic datasets. Using real-world scenes, we showcase the capability of our framework in producing photo-realistic scene enhancement. For downstream tasks, we use NICE data to finetune a vision-language model (VLM) for spatial affordance prediction and a vision-language-action (VLA) policy for object manipulation. Our evaluations show that NICE successfully minimizes OOD gaps, resulting in over 20% improvement in accuracy for affordance prediction in highly cluttered scenes. For manipulation tasks, success rate increases on average by 11% when testing in environments populated with distractors in different quantities. Furthermore, we show that our method improves visual robustness, lowering target confusion by 6%, and enhances safety by reducing collision rate by 7%.
BibTeX
@misc{pakdamansavoji2025improving,
  author = {Sajjad Pakdamansavoji and Mozhgan Pourkeshavarz and Adam Sigal and Zhiyuan Li and Rui Heng Yang and Amir Rasouli},
  title = {{Improving Robotic Manipulation Robustness via NICE Scene Surgery}},
  year = {2025},
  eprint = {2511.22777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.22777},
}
Figure