PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations

Haowen Sun, Han Wang, Chengzhong Ma, Shaolong Zhang, Jiawei Ye, Xingyu Chen, Xuguang LanarXiv 2025 · DOI · arXiv

Introduction

Training robots to perform robust manipulation tasks typically requires either massive amounts of real-world data or painstakingly hand-crafted simulation environments. PRISM (Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real) introduces a novel pipeline that bridges this gap. By utilizing just a handful of real-world demonstrations, PRISM automatically constructs a simulation environment and leverages Vision-Language Models (VLMs) to generate reward signals from simple spatial prompts, eliminating the need for manual reward engineering.

The Challenge of Robust Manipulation

Robotic manipulation faces a significant hurdle: generalization. Traditional Imitation Learning (e.g., Behavior Cloning) is intuitive but brittle; a robot trained on limited data often fails when encountering novel object poses. Conversely, Reinforcement Learning (RL) excels at exploration and robustness but is unsafe to perform in the real world. While simulation offers a safe training ground, the "sim-to-real gap" and the complexity of coding mathematical reward functions for every new task remain major bottlenecks.

PRISM addresses these issues by asking: How can we take just five real-world demonstrations, automatically build a training sandbox, and learn a robust RL policy without hand-coding a reward function?

The PRISM Pipeline

PRISM employs a three-step Real-to-Sim-to-Real methodology designed for maximum data efficiency and robustness.

1. Automated Real-to-Sim Transfer

Starting with a single real-world image and 5–15 expert demonstrations, PRISM uses foundation models like Segment Anything (SAM) and 6D pose estimation to identify objects. It then retrieves matching 3D models from a library to reconstruct the scene in a simulator. Real-world demonstrations are mapped into this virtual environment, with object poses refined to ensure physical consistency (e.g., preventing floating or clipping).

2. Projection-Based Reward Modeling

Instead of manual coding, PRISM uses "projection relationships" to define success. Humans provide simple text prompts describing how objects should appear from specific camera angles (e.g., "the gripper should overlap the apple").

PRISM pipeline overview showing the transition from real-world scene to multi-view simulation and VLM-based reward labeling.

As shown in the pipeline overview, a VLM (such as QWen-72B or GPT-4V) evaluates multi-view images generated by the simulator. It provides binary labels based on the spatial prompts, which are then used to train a lightweight reward model. This multi-view approach is critical; VLMs often struggle with single-view real-world images due to occlusion, but the simulator allows for consistent, multi-angle verification.

Multi-view simulation perspectives used to define spatial projection relationships for reward generation.

3. Sim-to-Real Co-Training and Safety

To bridge the domain shift back to reality, the policy is fine-tuned by mixing real-world demonstrations into the RL replay buffer. Furthermore, the learned reward model is repurposed as an "action feasibility predictor." Using single-view RGB-D data, this predictor acts as a real-time safety check, preventing the robot from executing potentially catastrophic actions if they don't align with the learned success criteria.

Key Results and Insights

PRISM was evaluated on a UR5e robotic arm across six diverse tasks, including lifting objects, pressing buttons, and placing items in bowls.

  • Superior Performance: PRISM achieved an 82% average success rate across tasks with randomized initial poses, using only 5 demonstrations. In contrast, standard Behavior Cloning (BC) achieved only 13% success under the same conditions.
  • Data Efficiency: Even when the BC baseline was provided with three times more data (15 demos), its success rate only reached 30%, highlighting the superior robustness of PRISM’s RL-based exploration.
  • VLM Accuracy: The study found that providing explicit geometric constraints ("does object A overlap object B?") rather than general success queries ("is the task done?") improved policy success rates by nearly 70%.
  • Safety Gains: Integrating the action feasibility predictor during deployment boosted real-world success rates by up to 37%.

Conclusion

PRISM demonstrates that highly robust robotic policies can be learned from minimal real-world data by automating the simulation and reward-labeling process. While currently dependent on the availability of 3D assets, this framework paves the way for fully automated, scene-aware training pipelines that can rapidly adapt to new manipulation tasks in complex environments.

Summary updated

Abstract

Learning from few demonstrations to develop policies robust to variations in robot initial positions and object poses is a problem of significant practical interest in robotics. Compared to imitation learning, which often struggles to generalize from limited samples, reinforcement learning (RL) can autonomously explore to obtain robust behaviors. Training RL agents through direct interaction with the real world is often impractical and unsafe, while building simulation environments requires extensive manual effort, such as designing scenes and crafting task-specific reward functions. To address these challenges, we propose an integrated real-to-sim-to-real pipeline that constructs simulation environments based on expert demonstrations by identifying scene objects from images and retrieving their corresponding 3D models from existing libraries. We introduce a projection-based reward model for RL policy training that is supervised by a vision-language model (VLM) using human-guided object projection relationships as prompts, with the policy further fine-tuned using expert demonstrations. In general, our work focuses on the construction of simulation environments and RL-based policy training, ultimately enabling the deployment of reliable robotic control policies in real-world scenarios.
BibTeX
@misc{sun2025prism,
  author = {Haowen Sun and Han Wang and Chengzhong Ma and Shaolong Zhang and Jiawei Ye and Xingyu Chen and Xuguang Lan},
  title = {{PRISM: Projection-based Reward Integration for Scene-Aware Real-to-Sim-to-Real Transfer with Few Demonstrations}},
  year = {2025},
  eprint = {2504.20520},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.20520},
}
Figure