Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation

Kenzo Lobos-Tsunekawa, Tatsuya Harada

Abstract

Reinforcement Learning (RL), among other learning-based methods, represents powerful tools to solve complex robotic tasks (e.g., actuation, manipulation, navigation, etc.), with the need for real-world data to train these systems as one of its most important limitations. The use of simulators is one way to address this issue, yet knowledge acquired in simulations does not work directly in the real-world, which is known as the sim-to-real transfer problem. While previous works focus on the nature of the images used as observations (e.g., textures and lighting), which has proven useful for a sim-to-sim transfer, they neglect other concerns regarding said observations, such as precise geometrical meanings, failing at robot-to-robot, and thus in sim-to-real transfers. We propose a method that learns on an observation space constructed by point clouds and environment randomization, generalizing among robots and simulators to achieve sim-to-real, while also addressing partial observability. We demonstrate the benefits of our methodology on the point goal navigation task, in which our method proves to be highly unaffected to unseen scenarios produced by robot-to-robot transfer, outperforms image-based baselines in robot-randomized experiments, and presents high performances in sim-to-sim conditions. Finally, we perform several experiments to validate the sim-to-real transfer to a physical domestic robot platform, confirming the out-of-the-box performance of our system.

Introduction

Deep Reinforcement Learning has shown massive potential for robotic tasks like visual navigation, but it requires millions of interactions to learn. Because training on physical robots is slow and dangerous, agents are usually trained in simulators. However, policies trained on 2D images (RGB or RGBD) usually fail in the real world due to the "sim-to-real gap."

Previous work has tried to close this gap by randomizing textures or using photo-realistic simulators. However, these methods often ignore the challenge of robot-to-robot transfer. If the simulated robot's camera height, angle, or field-of-view does not perfectly match the physical robot, the 2D image changes drastically, causing the policy to fail. Furthermore, robots suffer from partial observability—they can only see what is directly in front of them, making it easy to collide with obstacles they have already passed.

Methodology

To solve these issues, we propose a pipeline that transforms sensor data into a universal, robot-agnostic format before feeding it to the RL agent.

Point Cloud Observation Pipeline

Instead of feeding 2D depth images to a CNN, the depth data is projected into a 3D point cloud anchored to a fixed frame on the robot (e.g., its base). This makes the observation independent of the camera's specific placement or lens properties. To solve partial observability, the system uses an external registration system (like SLAM or visual odometry) to stitch point clouds from previous time steps into the current one. The robot builds a 3D memory of its surroundings. For computational efficiency, the aggregated point cloud is voxel-sampled and randomly downsampled to a fixed size of 1,024 points.

The proposed pipeline for transforming raw depth data into a canonical, integrated point cloud representation.

Reinforcement Learning Framework

The point cloud is processed by a 3D neural network (either PointNet or a modified ShellNet) to extract features, which are then fed into a standard Proximal Policy Optimization (PPO) algorithm. During training, the simulator randomizes camera parameters and injects noise into the robot's motion model to ensure the policy learns to be robust.

Experimental Setup

We utilized the Habitat simulation platform, training and evaluating on the MatterPort3D (MP3D) and Gibson datasets. For real-world validation, we used a physical Toyota HSR (Human Support Robot) equipped with depth sensors, using ORB-SLAM2 for real-time localization and point cloud integration.

Results

Our experiments demonstrate that point clouds serve as a "canonical space" that naturally solves the robot-to-robot transfer problem.

Robustness to Hardware Changes

When evaluated on a robot with different camera parameters than the one used in training, the standard RGBD baseline's success rate plummeted to 10%. In contrast, our proposed PointNet-based agent maintained an 85% success rate, proving its robustness to hardware changes.

Sim-to-Real Performance

Deployed on the physical Toyota HSR with zero fine-tuning, the point cloud agent successfully navigated to targets in approximately 75% of real-world trials. It successfully avoided complex obstacles like chair legs. In contrast, the RGBD baseline failed in almost all non-trivial real-world scenarios.

Visualization of a navigation trajectory within the integrated point cloud environment, demonstrating obstacle avoidance.

Safety and Memory

Explicitly integrating 3D point clouds over time proved more reliable than implicit memory (like RNNs) for understanding the immediate environment. This drastically reduced collisions with unseen obstacles. We also modified the training environment to terminate episodes upon collision, forcing the agent to learn genuinely safe trajectories.

Conclusion and Future Work

This research confirms that 3D point cloud-based RL offers a robust solution for sim-to-real transfer in visual navigation. By creating a robot-agnostic observation space, we can train policies in simulation that generalize to physical hardware out-of-the-box. Future work will address the reliance on external SLAM systems and explore more complex tasks like active mapping and semantic instruction following.

Summary updated

Abstract

Reinforcement Learning (RL), among other learning-based methods, represents powerful tools to solve complex robotic tasks (e.g., actuation, manipulation, navigation, etc.), with the need for real-world data to train these systems as one of its most important limitations. The use of simulators is one way to address this issue, yet knowledge acquired in simulations does not work directly in the real-world, which is known as the sim-to-real transfer problem. While previous works focus on the nature of the images used as observations (e.g., textures and lighting), which has proven useful for a sim-to-sim transfer, they neglect other concerns regarding said observations, such as precise geometrical meanings, failing at robot-to-robot, and thus in sim-to-real transfers. We propose a method that learns on an observation space constructed by point clouds and environment randomization, generalizing among robots and simulators to achieve sim-to-real, while also addressing partial observability. We demonstrate the benefits of our methodology on the point goal navigation task, in which our method proves to be highly unaffected to unseen scenarios produced by robot-to-robot transfer, outperforms image-based baselines in robot-randomized experiments, and presents high performances in sim-to-sim conditions. Finally, we perform several experiments to validate the sim-to-real transfer to a physical domestic robot platform, confirming the out-of-the-box performance of our system.
BibTeX
@inproceedings{lobostsunekawa2020point,
  author = {Kenzo Lobos-Tsunekawa and Tatsuya Harada},
  title = {{Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=XIpRiXI752BV},
}
Figure