Abstract
For a robot to learn a good policy, it often requires expensive equipment (such as sophisticated sensors) and a prepared training environment conducive to learning. However, it is seldom possible to perfectly equip robots for economic reasons, nor to guarantee ideal learning conditions, when deployed in real-life environments. A solution would be to prepare the robot in the lab environment, when all necessary material is available to learn a good policy. After training in the lab, the robot should be able to get by without the expensive equipment that used to be available to it, and yet still be guaranteed to perform well on the field. The transition between the lab (source) and the real-world environment (target) is related to transfer learning, where the state-space between the source and target tasks differ. We tackle a simulated task with continuous states and discrete actions presenting this challenge, using Bootstrapped Dual Policy Iteration, a model-free actor-critic reinforcement learning algorithm, and Policy Shaping. Specifically, we train a BDPI agent, embodied by a virtual robot performing a task in the V-Rep simulator, sensing its environment through several proximity sensors. The resulting policy is then used by a second agent learning the same task in the same environment, but with camera images as input. The goal is to obtain a policy able to perform the task relying on merely camera images.
Introduction
In robotics, a significant gap exists between controlled laboratory settings and real-world deployments. While lab robots can be outfitted with high-fidelity sensors like LIDAR or dense proximity arrays, consumer or industrial robots often rely on more economical, lower-fidelity sensors such as a single camera. Learning navigation directly from raw camera pixels is a high-dimensional, complex Reinforcement Learning (RL) problem. Conversely, learning from proximity sensors is relatively simple due to the low-dimensional, direct spatial data. This research addresses cross-sensor transfer learning: leveraging a policy learned via "easy" sensors to accelerate the training of a robot equipped only with "hard" sensors.
Methodology
The study employs two core frameworks:
- Bootstrapped Dual Policy Iteration (BDPI): A sample-efficient actor-critic RL algorithm utilizing one actor and multiple critics to stabilize and speed up learning.
- Policy Shaping (Actor-Advisor): A mechanism where a pre-trained "advisor" (the proximity-sensor policy) guides a "student" (the camera-based robot).
Three transfer strategies were evaluated: Transfer at Acting Time (biasing exploration), Transfer at Learning Time (active imitation), and a Mixed Approach.
Experimental Setup
Experiments were conducted in the V-REP simulator using a simulated Epuck robot. The environment consisted of a square arena with an off-center pillar. The "Source" task utilized 8 proximity sensors, while the "Target" task relied on raw images from a front-facing camera. The action space included 5 discrete movements: stay still, turn left, turn right, move forward, and decelerate. During training, the simulator provided both sensor streams, allowing the advisor to process proximity data while the student learned to map camera images to successful actions.
Results and Discussion
The primary bottleneck in learning from complex sensors is exploration. Results indicate that transferring at acting time is the most effective strategy. By guiding the robot's exploration, the student quickly learns the relationship between camera pixels and navigation.
As shown in the performance graph, the "no transfer" baseline struggled to learn a viable policy. In contrast, the transfer-act1-learn0.0 configuration (acting-time transfer only) achieved the highest rewards and fastest convergence. Interestingly, aggressive imitation ("learning time" transfer) proved less effective, potentially because it hindered the student's ability to develop its own sensor-to-action mapping. Notably, this method requires no explicit translation function between sensor types, as the dual-sensor availability during training sidesteps the mapping problem.
Conclusion
This research demonstrates that cross-sensor transfer learning via policy shaping significantly improves the feasibility of training robots with complex sensors. While the method requires a "lab" phase where both sensor types are available, it allows for the deployment of high-performing policies on robots with limited hardware. Future work will focus on optimizing the balance between acting-time and learning-time transfer and extending the approach to more complex, multi-task environments.
Summary updated