Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computers

Yan Wang, Gautham Vasan, A. Rupam MahmoodICRA 2022 · DOI · OpenReview

TL;DR

This paper introduces ReLoD, a system for distributing real-time reinforcement learning computations between a local, resource-limited robot computer and a powerful remote workstation. They demonstrate that simply offloading all computation to a remote machine doesn't guarantee performance gains for complex algorithms like SAC, but a carefully balanced distribution can significantly improve performance.

What Problem Does This Paper Address?

Real-time learning is essential for robots to adapt to dynamic environments. A common setup involves a robot with a limited onboard computer and a powerful, wirelessly connected workstation. However, it's unclear how best to leverage this distributed computing setup. Specifically, how do resource limitations on the local computer affect learning performance, and how can a powerful remote computer be used effectively to compensate for these limitations without introducing new bottlenecks? This paper tackles this by investigating different computational distributions for real-time reinforcement learning (RL) on robots.

Methods

The authors propose the Remote-Local Distributed (ReLoD) system, which allows for distributing the computationally intensive components of RL algorithms (action computation, policy updating, and minibatch sampling) between a local and a remote computer. ReLoD supports three modes:

  • Remote-Only: All computations happen on the remote computer.
  • Local-Only: All computations happen on the local computer.
  • Remote-Local: A hybrid approach where computations are carefully distributed.

They implement and evaluate two popular deep RL algorithms, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), within the ReLoD framework. The system is designed to be modular, allowing for the integration of different RL algorithms.

Datasets

The paper introduces two new vision-based robotic tasks for evaluation:

  • UR5-VisualReacher: A UR5 robotic arm attempts to move its fingertip to a target on a monitor, using joint velocities as actions and a combination of joint states, velocities, and stacked images as observations.
  • Create-Reacher: An iRobot Create2 mobile robot navigates to a target on a wall, using wheel speeds as actions and wall sensor readings and stacked images as observations.

These tasks are designed to be challenging, requiring significant visual perception and control, and are evaluated using real-world robot interactions.

Key Insights and Ideas

  • Remote-Only isn't always better: Offloading all computations to a powerful remote computer doesn't automatically lead to improved performance for complex algorithms like SAC. This is likely due to the increased latency introduced by wireless communication for action commands.
  • The "sweet spot" is Remote-Local: A carefully balanced distribution of computations, where local action computation is paired with remote policy updates, significantly boosts SAC's performance on resource-constrained robots.
  • Algorithm matters: Computationally intensive algorithms like SAC benefit greatly from ReLoD's distributed approach, whereas simpler algorithms like PPO are less affected by the distribution strategy.
  • Resource limitations are significant: SAC's performance degrades substantially when confined to a resource-limited local computer.
  • ReLoD is a viable alternative: The Local-Only mode of ReLoD performs comparably to existing, well-tuned single-computer RL systems, making it suitable for conventional research.

Key Results

  • On the UR5-VisualReacher task, SAC's performance on a resource-limited laptop dropped by approximately 28% compared to a powerful workstation.
  • The Remote-Only mode for SAC on UR5-VisualReacher showed little improvement over the Local-Only mode and exhibited higher variance.
  • The Remote-Local mode for SAC consistently improved performance, achieving over 90% of the performance of the best-case Local-WStation-Only configuration.
  • For PPO, performance was largely unaffected across the three distribution modes on UR5-VisualReacher.
  • SAC significantly outperformed PPO on the complex vision-based tasks, achieving substantially higher average returns.
  • The Local-Only mode of ReLoD on a powerful workstation performed on par with a state-of-the-art asynchronous SAC implementation.

Limitations and Future Work

  • Latency impact: The paper highlights the detrimental effect of latency, particularly when it occurs closer to robot actuation. While ReLoD mitigates this by keeping action computation local in the Remote-Local mode, the Remote-Only mode is more susceptible.
  • Algorithm dependency: The benefits of ReLoD are more pronounced for computationally intensive algorithms like SAC. Future work could explore optimal distribution strategies for a wider range of RL algorithms.
  • Task-specific tuning: While ReLoD is presented as task-agnostic, the optimal distribution might still require some tuning based on the specific robot and task.
  • Future work: The authors suggest that ReLoD provides a valuable tool for real-time RL research, particularly for computationally demanding algorithms. They also note that their Local-Only mode is suitable for conventional RL research, offering a comparable performance to existing single-computer systems.

Summary updated

Abstract

Real-time learning is crucial for robotic agents adapting to ever-changing, non-stationary environments. A common setup for a robotic agent is to have two different computers simultaneously: a resource-limited local computer tethered to the robot and a powerful remote computer connected wirelessly. Given such a setup, it is unclear to what extent the performance of a learning system can be affected by resource limitations and how to efficiently use the wirelessly connected powerful computer to compensate for any performance loss. In this paper, we implement a real-time learning system called the Remote-Local Distributed (ReLoD) system to distribute computations of two deep reinforcement learning (RL) algorithms, Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), between a local and a remote computer. The performance of the system is evaluated on two vision-based control tasks developed using a robotic arm and a mobile robot. Our results show that SAC's performance degrades heavily on a resource-limited local computer. Strikingly, when all computations of the learning system are deployed on a remote workstation, SAC fails to compensate for the performance loss, indicating that, without careful consideration, using a powerful remote computer may not result in performance improvement. However, a carefully chosen distribution of computations of SAC consistently and substantially improves its performance on both tasks. On the other hand, the performance of PPO remains largely unaffected by the distribution of computations. In addition, when all computations happen solely on a powerful tethered computer, the performance of our system remains on par with an existing system that is well-tuned for using a single machine. ReLoD is the only publicly available system for real-time RL that applies to multiple robots for vision-based tasks. The source code can be found at https://github.com/rlai-lab/relod
BibTeX
@misc{wang2022real,
  author = {Yan Wang and Gautham Vasan and A. Rupam Mahmood},
  title = {{Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computers}},
  year = {2022},
  eprint = {2210.02317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.02317},
}
Figure