RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation

Xiangyu Chen, Chuhao Zhou, Yuxi Liu, Jianfei YangarXiv 2025 · DOI · arXiv

Introduction

Achieving millimeter-level precision in robotic manipulation remains a significant challenge in automation. Tasks such as reagent handling in laboratories require sub-millimeter accuracy, which traditional Imitation Learning (IL) and Reinforcement Learning (RL) struggle to provide. IL is often limited by the difficulty of obtaining consistent high-precision human demonstrations via teleoperation. Meanwhile, RL typically suffers from simulation-to-real gaps, extreme data inefficiency in physical environments, and instabilities caused by distribution shifts when reusing historical data.

The Role-Model Reinforcement Learning Framework

To address these challenges, we introduce Role-Model Reinforcement Learning (RM-RL). This framework enables a robot to act as its own teacher by identifying the most successful actions from its own online trials and using them to supervise its learning process.

Methodology

The RM-RL process transforms the standard RL problem into a stable supervised learning task through three primary components:

  1. Role-Model Online Labeling: During trials within a specific "scene" (initial condition), the system identifies the action that yielded the highest reward. This action is designated as the "Role-Model."
  2. Supervised Reformulation: Instead of relying on high-variance policy gradients, the system treats the Role-Model action as a ground-truth label for all other actions in that scene. This allows the use of stable cross-entropy losses.
  3. Data Replay: Labeled data is stored in a replay buffer and reused multiple times, ensuring that every expensive real-world interaction contributes to policy improvement repeatedly.
The RM-RL workflow: Online training is stabilized by labeling the best-performing actions as 'Role-Models' for supervised learning and data replay.

The underlying policy architecture utilizes a ResNet backbone for visual processing and an MLP for robot pose encoding, outputting discrete probability distributions for precise adjustments in translation ($x, y$) and rotation ($\psi$).

Experimental Results

The framework was evaluated using a Ufactory X-ARM 6 robot on high-precision tasks, including picking cell plates and placing them into narrow shelf slots.

Performance and Convergence

RM-RL demonstrated significant improvements over standard RL baselines in both accuracy and training efficiency. Specifically, it achieved a 53% improvement in translation accuracy (reducing error to 2.1mm) and a 20% improvement in rotation accuracy.

Reward curves comparing RM-RL against standard RL baselines, demonstrating faster convergence and superior stability.

In the shelf-placement task, the pretrained RM-RL model achieved a 100% success rate, doubling the 50% success rate of standard RL. The system leverages zero-shot foundation models (Grounding DINO and SAM) for initial localization, while the RM-RL policy handles the critical "last mile" of precision that vision models alone often miss.

Conclusion

RM-RL offers a robust solution for high-precision robotic tasks by stabilizing real-world RL through supervised reformulation. By treating the best local action as a "role model," the system creates a stable target for optimization without human intervention. While currently focused on short-horizon adjustments, future work will explore extending this strategy to complex, multi-stage sequential tasks.

Summary updated

Abstract

Precise robot manipulation is critical for fine-grained applications such as chemical and biological experiments, where even small errors (e.g., reagent spillage) can invalidate an entire task. Existing approaches often rely on pre-collected expert demonstrations and train policies via imitation learning (IL) or offline reinforcement learning (RL). However, obtaining high-quality demonstrations for precision tasks is difficult and time-consuming, while offline RL commonly suffers from distribution shifts and low data efficiency. We introduce a Role-Model Reinforcement Learning (RM-RL) framework that unifies online and offline training in real-world environments. The key idea is a role-model strategy that automatically generates labels for online training data using approximately optimal actions, eliminating the need for human demonstrations. RM-RL reformulates policy learning as supervised training, reducing instability from distribution mismatch and improving efficiency. A hybrid training scheme further leverages online role-model data for offline reuse, enhancing data efficiency through repeated sampling. Extensive experiments show that RM-RL converges faster and more stably than existing RL methods, yielding significant gains in real-world manipulation: 53% improvement in translation accuracy and 20% in rotation accuracy. Finally, we demonstrate the successful execution of a challenging task, precisely placing a cell plate onto a shelf, highlighting the framework's effectiveness where prior methods fail.
BibTeX
@misc{chen2025rm,
  author = {Xiangyu Chen and Chuhao Zhou and Yuxi Liu and Jianfei Yang},
  title = {{RM-RL: Role-Model Reinforcement Learning for Precise Robot Manipulation}},
  year = {2025},
  eprint = {2510.15189},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15189},
}
Figure