Bayesian Multi-Task Reinforcement Learning

Alessandro Lazaric, Mohammad GhavamzadehICML 2010 · OpenReview

Introduction

Reinforcement Learning (RL) is notoriously data-hungry. Standard algorithms often require thousands of interactions to learn an accurate value function—an estimate of how "good" a given state is. In many real-world scenarios, agents are faced with multiple different tasks but can only generate a handful of samples for each.

While Multi-Task Learning (MTL) aims to solve this data scarcity by sharing information across tasks, it faces two major hurdles:

  1. What to share? Most methods share transition dynamics or reward structures. This work instead focuses on sharing the structure of the value function itself.
  2. Negative Transfer: If an agent tries to learn from a "related" task that is actually fundamentally different, its performance can degrade. The challenge lies in identifying which tasks belong together without prior labeling.

This paper introduces a hierarchical Bayesian framework for multi-task RL that allows agents to share information across related tasks by modeling similarities in their value functions. By using Dirichlet Process priors to automatically cluster tasks into "families," the proposed algorithms significantly improve learning efficiency in data-scarce environments while avoiding the pitfalls of negative transfer.

Methods

The authors build their approach on Gaussian Process Temporal-Difference (GPTD) models. They treat the weight vector of a task's value function as a random variable drawn from a prior distribution.

To handle different degrees of task relatedness, they propose two main models:

  • SCMTL (Single-Class Multi-Task Learning): This model assumes all tasks are related and share a single common prior (a Normal-Inverse-Wishart distribution). It uses an Expectation-Maximization (EM) algorithm to find the best parameters for this shared prior.
  • MCMTL (Multi-Class Multi-Task Learning): This is a more advanced, non-parametric version. It uses a Dirichlet Process (DP) prior, which allows the model to automatically discover an undefined number of task clusters. If a new task doesn't fit into existing clusters, the DP allows the model to create a new one. Inference is handled via a combination of Gibbs sampling and EM.
Graphical models for STL, SCMTL, and MCMTL

The paper evaluates these models in two settings: Symmetric (learning all tasks simultaneously) and Asymmetric (using knowledge from previously learned tasks to speed up learning on a brand-new task).

Experimental Setup

The authors evaluated their algorithms on two primary benchmarks:

  1. Synthetic Regression Problem: A controlled environment where tasks are functions in a linear space. Weights were drawn from four distinct 6-dimensional Gaussian distributions to test the model's ability to cluster tasks correctly.
  2. Inverted Pendulum: A classic RL benchmark. Here, "tasks" were created by varying the physical parameters of the system—such as the mass of the pole, the length of the pole, and the mass of the cart—drawn from two different classes of dynamics.

Key Results and Insights

By sharing the value function structure rather than just raw data or dynamics, the agent leverages similarities even when the underlying physics of the tasks differ slightly. The use of a DP prior serves as the key mechanism for avoiding negative transfer; instead of forcing a "one-size-fits-all" prior, the model learns a taxonomy of the tasks it faces.

Synthetic regression task parameters and performance heatmap
  • Massive Sample Efficiency: In the synthetic regression task, SCMTL improved performance over baseline Single-Task Learning (STL) by up to 67.6% when only 20 samples per task were available.
  • Clustering Accuracy: MCMTL outperformed SCMTL significantly when tasks came from different distributions, improving Mean Squared Error (MSE) by up to 77.6% compared to STL. This proves that MCMTL successfully separates unrelated tasks to avoid negative transfer.
  • RL Performance: On the Inverted Pendulum, both Bayesian methods reached high performance levels with far fewer samples than STL. While STL required roughly 1,200 samples to converge, the multi-task models showed significant progress with just 100–500 samples.
  • Efficiency in the Asymmetric Setting: The model demonstrates that once the "classes" of tasks are learned, a new task can be mastered with a fraction of the usual data by simply identifying its "family" and adopting that family's prior.

Limitations and Future Work

While highly effective, the framework has a few limitations. First, the use of Gibbs sampling in the MCMTL algorithm can become computationally expensive as the number of tasks and clusters grows. Second, the current model assumes all tasks share the same state/action spaces and the same set of basis functions (features). Extending this to tasks with different feature representations remains an open challenge. Finally, the paper focuses primarily on policy evaluation (learning the value of a fixed policy). Integrating this seamlessly into a full policy iteration loop—where the policy changes at every step—is a complex but necessary next step, as changing policies can alter how tasks cluster together.

Summary updated

Abstract

We consider the problem of multi-task reinforcement learning where the learner is provided with a set of tasks, for which only a small number of samples can be generated for any given policy. As the number of samples may not be enough to learn an accurate evaluation of the policy, it would be necessary to identify classes of tasks with similar structure and to learn them jointly. We consider the case where the tasks share structure in their value functions, and model this by assuming that the value functions are all sampled from a common prior. We adopt the Gaussian process temporal-difference value function model and use a hierarchical Bayesian approach to model the distribution over the value functions. We study two cases, where all the value functions belong to the same class and where they belong to an undefined number of classes. For each case, we present a hierarchical Bayesian model, and derive inference algorithms for (i) joint learning of the value functions, and (ii) efficient transfer of the information gained in (i) to assist learning the value function of a newly observed task.
BibTeX
@inproceedings{lazaric2010bayesian,
  author = {Alessandro Lazaric and Mohammad Ghavamzadeh},
  title = {{Bayesian Multi-Task Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2010},
  url = {https://openreview.net/forum?id=rJb3V3ZOWr},
}
Figure