Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation

Kenzo Lobos-Tsunekawa, Tatsuya Harada

TL;DR

Training Reinforcement Learning (RL) agents in simulation and deploying them on physical robots often fails due to the "sim-to-real" gap. This paper tackles the issue by ditching standard 2D image observations in favor of 3D point clouds, creating a "canonical" observation space that is invariant to specific camera parameters. This approach enables impressive out-of-the-box sim-to-real transfer and handles partial observability by explicitly stitching together past and present 3D observations.

What Problem Does This Paper Address?

Deep Reinforcement Learning has shown massive potential for robotic tasks like visual navigation, but it requires millions of interactions to learn. Because training on physical robots is slow and dangerous, agents are usually trained in simulators. However, policies trained on 2D images (RGB or RGBD) usually fail in the real world. This is the sim-to-real gap.

Previous work has tried to close this gap by randomizing textures or using photo-realistic simulators. But these methods ignore a glaring issue: robot-to-robot transfer. If the simulated robot's camera height, angle, or field-of-view (intrinsics and extrinsics) doesn't perfectly match the physical robot, the 2D image changes drastically, and the policy breaks. Furthermore, robots suffer from partial observability—they can only see what is directly in front of them, making it easy to forget obstacles they just passed.

Methods

To solve these issues, the authors propose a pipeline that transforms sensor data into a universal, robot-agnostic format before feeding it to the RL agent:

  1. Point Cloud Projection: Instead of feeding 2D depth images to a CNN, the depth data is projected into a 3D point cloud anchored to a fixed frame on the robot (e.g., its base). This makes the observation independent of the camera's specific placement or lens properties.
  2. Temporal Integration: To solve partial observability, the system uses an external registration system (like SLAM or visual odometry) to stitch point clouds from previous time steps into the current one. The robot literally builds a 3D memory of its surroundings.
  3. Downsampling: For computational efficiency, the aggregated point cloud is voxel-sampled and randomly downsampled to a fixed size of 1,024 points.
  4. RL Architecture: The point cloud is processed by a 3D neural network (either PointNet or a modified ShellNet) to extract features, which are then fed into a standard Proximal Policy Optimization (PPO) algorithm.
  5. Domain Randomization: During training, the simulator randomizes camera parameters and injects noise into the robot's motion model to ensure the policy learns to be robust.

Datasets

  • Simulators & Environments: The team used the Habitat simulation platform, training and evaluating on the MatterPort3D (MP3D) and Gibson datasets (which feature 3D scans of real-world indoor environments).
  • Real-World Hardware: A physical Toyota HSR (Human Support Robot) equipped with depth sensors, using ORB-SLAM2 for real-time localization and point cloud integration.

Key Insights and Ideas

  • Point Clouds as a "Canonical Space": By projecting depth data into 3D space relative to the robot's base, the observation becomes purely geometric. A wall is a wall, regardless of whether the camera is mounted 1 meter or 1.5 meters high. This naturally solves the robot-to-robot transfer problem.
  • Explicit Memory Beats Implicit Memory: Standard RL approaches use Recurrent Neural Networks (RNNs) to "remember" past 2D frames. The authors show that explicitly integrating 3D point clouds over time is a much more reliable way to give the agent a complete understanding of its immediate environment, drastically reducing collisions with unseen obstacles.
  • Safety-Aware Metrics Matter: The authors noted that standard RL navigation benchmarks don't heavily penalize collisions. They modified the training environment to terminate episodes upon collision, forcing the agent to learn genuinely safe trajectories suitable for the real world.

Key Results

  • Robot-to-Robot Transfer (Simulation): When evaluated on a robot with different camera parameters than the one used in training, the standard RGBD baseline's success rate plummeted to 10%. The proposed PointNet-based agent maintained an 85% success rate, proving its robustness to hardware changes.
  • Sim-to-Sim Transfer: The point cloud agent showed superior generalization when trained in one dataset (MP3D) and evaluated in a completely unseen one (Gibson), suffering less performance degradation than the image-based baseline.
  • Sim-to-Real Transfer: Deployed on the physical Toyota HSR with zero fine-tuning, the point cloud agent successfully navigated to targets in about 75% of real-world trials. It successfully avoided complex obstacles like chair legs. In contrast, the RGBD baseline failed in almost all non-trivial real-world scenarios.

Limitations and Future Work

  • External Dependencies: The method relies heavily on an external system (like ORB-SLAM2) to track the robot's position and stitch point clouds together. If the SLAM system fails or drifts, the RL agent's observation space corrupts.
  • Computational Constraints: Processing 3D point clouds is memory-intensive. The authors had to limit the input to just 1,024 points, which prevents the use of heavier, potentially more powerful modern 3D network architectures.
  • Unseen Object Collisions: The agent still occasionally collided with objects it never directly observed, suggesting that pure RL signals might not be enough to teach "common sense" physics or object permanence.
  • Future Work: The authors suggest applying this canonical point cloud approach to more complex tasks like active mapping, active vision, and semantic instruction following.

Summary updated

Abstract

Reinforcement Learning (RL), among other learning-based methods, represents powerful tools to solve complex robotic tasks (e.g., actuation, manipulation, navigation, etc.), with the need for real-world data to train these systems as one of its most important limitations. The use of simulators is one way to address this issue, yet knowledge acquired in simulations does not work directly in the real-world, which is known as the sim-to-real transfer problem. While previous works focus on the nature of the images used as observations (e.g., textures and lighting), which has proven useful for a sim-to-sim transfer, they neglect other concerns regarding said observations, such as precise geometrical meanings, failing at robot-to-robot, and thus in sim-to-real transfers. We propose a method that learns on an observation space constructed by point clouds and environment randomization, generalizing among robots and simulators to achieve sim-to-real, while also addressing partial observability. We demonstrate the benefits of our methodology on the point goal navigation task, in which our method proves to be highly unaffected to unseen scenarios produced by robot-to-robot transfer, outperforms image-based baselines in robot-randomized experiments, and presents high performances in sim-to-sim conditions. Finally, we perform several experiments to validate the sim-to-real transfer to a physical domestic robot platform, confirming the out-of-the-box performance of our system.
BibTeX
@inproceedings{lobostsunekawa2020point,
  author = {Kenzo Lobos-Tsunekawa and Tatsuya Harada},
  title = {{Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=XIpRiXI752BV},
}

Loading related content...

Figure