Transfer Reinforcement Learning with Shared Dynamics

Romain Laroche, Merwan BarlierAAAI 2017 · OpenReview

Abstract: This article addresses a particular Transfer Reinforcement Learning (RL) problem: when dynamics do not change from one task to another, and only the reward function does. Our method relies on two ideas, the first one is that transition samples obtained from a task can be reused to learn on any other task: an immediate reward estimator is learnt in a supervised fashion and for each sample, the reward entry is changed by its reward estimate. The second idea consists in adopting the optimism in the face of uncertainty principle and to use upper bound reward estimates. Our method is tested on a navigation task, under four Transfer RL experimental settings: with a known reward function, with strong and weak expert knowledge on the reward function, and with a completely unknown reward function. It is also evaluated in a Multi-Task RL experiment and compared with the state-of-the-art algorithms. Results reveal that this method constitutes a major improvement for transfer/multi-task problems that share dynamics.

Introduction

Reinforcement Learning (RL) agents typically require a massive amount of experience to learn a task from scratch. In many real-world scenarios, such as robotics, an agent might need to perform many different tasks in the same physical environment. While the goal changes (e.g., "pick up the red ball" vs. "open the blue door"), the underlying laws of physics—the dynamics—remain the same. Standard RL often ignores this shared structure, relearning how the world works every time a new goal is introduced. This paper introduces a method for Transfer Reinforcement Learning with Shared Dynamics (TRLSD), focusing on how to efficiently reuse knowledge of "how the world moves" to accelerate learning in new reward environments.

Methodology

The proposed framework integrates two core concepts into a "Transition Reuse" system using Fitted-Q Iteration (FQI), a batch RL algorithm well-suited for stored data.

Transition Sample Sharing

Since the dynamics ($P$) are shared across tasks, any $(s, a, s')$ transition tuple recorded in Task A remains valid for Task B. The agent maintains a global library of these transitions. To learn a new Task $\tau$, the agent estimates a reward function $\hat{R}_\tau$ and "projects" the entire library onto the new task by replacing the old rewards with these new estimates. This allows the agent to "hallucinate" experience for a new task without physical re-exploration.

Optimism in Continuous Spaces

To guide exploration effectively, the authors adapt the "Optimism in the Face of Uncertainty" (UCRL) principle for continuous state spaces. They propose a Gaussian similarity measure to calculate a "neighbouring weight" for any state based on the density of previously seen samples. This weight defines a confidence interval, allowing the agent to target areas with high potential rewards or high uncertainty (low sample density).

Visualizing density-based optimism for exploration in continuous state spaces.

This density-based approach prevents "premature exploitation," where an agent stops searching after finding a sub-optimal reward. By forcing the agent to explore unknown parts of the map, the system ensures more robust convergence.

Experimental Evaluation

The approach was evaluated on a 2D Navigation Maze, a $5 \times 5$ environment where the agent has continuous coordinates but moves in four cardinal directions with added noise. The study utilized 25 distinct tasks, each with a reward hidden in a different cell.

Learned policy and agent trajectory in the 2D navigation maze.

Experiments covered various levels of prior knowledge, from completely unknown rewards to "expert knowledge" scenarios where rewards were known to be located in specific areas like corners.

Key Results and Insights

  • Significant Speedup: In Multi-Task settings, the agent nearly solved 25 tasks after only 600 transitions per task. In contrast, non-transfer benchmarks failed to learn basic policies even with 1,000 transitions per task.
  • Continuous Advantage: The continuous version of the algorithm (using density-based confidence intervals) often outperformed discrete versions, likely due to more frequent policy updates triggered as the agent moved through the continuous space.
  • Representation Decoupling: A critical insight is that the state representation for the policy (continuous features) can differ from the representation used for reward estimation. This allows designers to inject discrete expert knowledge into the reward estimator while maintaining a continuous policy.

Limitations and Future Work

While effective, the method currently assumes exact dynamics across tasks. Real-world shifts, such as mechanical wear in a robot, are not yet handled. Additionally, the performance is sensitive to hyperparameters like the optimism parameter ($\kappa$) and similarity scale ($\sigma$). Future work will focus on scaling the computational efficiency of calculating neighbouring weights as the transition library grows and adapting the framework for environments with slightly shifting dynamics.

Summary updated

Abstract

This article addresses a particular Transfer Reinforcement Learning (RL) problem: when dynamics do not change from one task to another, and only the reward function does. Our method relies on two ideas, the first one is that transition samples obtained from a task can be reused to learn on any other task: an immediate reward estimator is learnt in a supervised fashion and for each sample, the reward entry is changed by its reward estimate. The second idea consists in adopting the optimism in the face of uncertainty principle and to use upper bound reward estimates. Our method is tested on a navigation task, under four Transfer RL experimental settings: with a known reward function, with strong and weak expert knowledge on the reward function, and with a completely unknown reward function. It is also evaluated in a Multi-Task RL experiment and compared with the state-of-the-art algorithms. Results reveal that this method constitutes a major improvement for transfer/multi-task problems that share dynamics.
BibTeX
@inproceedings{laroche2017transfer,
  author = {Romain Laroche and Merwan Barlier},
  title = {{Transfer Reinforcement Learning with Shared Dynamics}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=SkEJ36gdZH},
}
Figure