Learning Probabilistic Relational Dynamics for Multiple Tasks

Ashwin Deshpande, Brian Milch, Luke S. Zettlemoyer, Leslie Pack Kaelbling

TL;DR

This paper introduces a hierarchical Bayesian approach to learning probabilistic rules that describe how actions change the world, specifically for multiple related tasks. By learning a shared "prototype" rule set, the system can effectively transfer knowledge between tasks, significantly reducing the amount of data needed to learn new task dynamics.

What Problem Does This Paper Address?

Intelligent agents need to understand how their actions affect the environment to plan effectively. This knowledge is often encoded as probabilistic planning rules. While algorithms exist to learn these rules from data, they typically learn each task from scratch. This is inefficient, as humans are adept at transferring knowledge from familiar situations to new, related ones. The paper addresses this gap by developing a method for transfer learning in the context of learning relational probabilistic planning rules, aiming to leverage data from multiple related tasks to improve learning on a new, unseen task.

Methods

The core of the paper is a hierarchical Bayesian model. This model treats task-specific rule sets (R1, ..., RK) as samples from a global prior distribution (G). Crucially, this global prior, or "rule set prototype," is learned from the data, allowing the system to discover commonalities across tasks.

The learning process involves two main stages:

  1. Learning the Rule Set Prototype (G): The algorithm first learns the best rule set prototype G from the data of K source tasks. This is done using a coordinate ascent algorithm* that iteratively optimizes G and the individual task rule sets.
  2. Learning the Target Task Rule Set (RK+1): With G* fixed, the algorithm then learns the rule set for the new target task using the observed data for that task, biased by the learned prototype.

Key novelties include:

  • Rule Set Prototypes: Instead of a fixed prior, the model learns a "prototype" rule set that captures shared structure. This prototype is then stochastically modified to generate task-specific rule sets.
  • Hierarchical Bayesian Framework: This allows for principled integration of information across tasks and provides a probabilistic framework for learning.
  • Coordinate Ascent Algorithm: An iterative approach to optimize both the global prototype and local task-specific rules.
  • Specialized Operators: The learning algorithm uses operators for adding/removing rules and literals, splitting rules, and importantly, learning outcomes for rules.

Datasets

The paper evaluates its approach on synthetic data generated from four families of related tasks within the blocks world domain. These domains are variants of the classic blocks world, focusing on the effects of a single action, pickup(X, Y). The domains differ in complexity, including:

  • Gripper Size Domain: Tasks vary in the size of the gripper, requiring rules with different size literals in their contexts.
  • Slippery Gripper Domain: Tasks involve variations in outcome probabilities for a fixed set of rules.
  • Slippery Gripper with Size Domain: Combines structural and parametric variations.
  • Random Domain: Used to test for erroneous transfer when tasks are unrelated, with randomly generated rule sets.

The methodology involves generating K source tasks, training data for each, learning a prototype, generating a target task and its data, learning the target rule set using the prototype, and finally evaluating performance on a test set.

Key Insights and Ideas

  • Structure Matters: The success of transfer learning hinges on learning not just the probabilistic parameters but also the underlying relational structure of the rules.
  • Shared Prototypes Capture Commonality: A learned rule set prototype effectively captures commonalities across related tasks, providing a strong inductive bias for learning new tasks.
  • Hierarchical Models Enable Transfer: The hierarchical Bayesian framework is a natural fit for multi-task learning, allowing information to flow from individual tasks up to the global model and back down.
  • Avoiding Erroneous Transfer: The experiments show that when tasks are truly unrelated, the learned prototype does not negatively impact performance, indicating robustness.
  • Data Efficiency: Transfer learning significantly reduces the number of training examples required to achieve high accuracy on a new task.

Key Results

The paper demonstrates significant improvements in learning efficiency across various blocks world domains. For instance, in the "Gripper Size Domain," transfer learning consistently achieved higher accuracy with fewer target task examples compared to learning from scratch. Similar gains were observed in the "Slippery Gripper" and "Slippery Gripper with Size" domains, highlighting the benefits of both parametric and structural transfer. In the "Random Domain," where tasks were unrelated, the transfer and non-transfer learning approaches performed statistically indistinguishably, showing that the algorithm doesn't harm performance when transfer is not beneficial.

Limitations and Future Work

  • Formula Distribution: The authors note that their current distribution over formulas might be "overly flat," suggesting that a more sophisticated distribution that favors common literals or subformulas could improve performance.
  • Computational Cost: While the coordinate ascent algorithm is used, the search for optimal rule sets and prototypes can still be computationally intensive.
  • Generalization to More Complex Domains: The current work focuses on a single action and relatively simple blocks world scenarios. Extending the approach to more complex domains with multiple interacting actions and richer representations is a natural next step.
  • Broader Representations: The authors suggest that the framework could be extended to work with a wider range of representation languages beyond the specific probabilistic relational rules used here.

Summary updated

Abstract

The ways in which an agent's actions affect the world can often be modeled compactly using a set of relational probabilistic planning rules. This extended abstract addresses the problem of learning such rule sets for multiple related tasks. We take a hierarchical Bayesian approach, in which the system learns a prior distribution over rule sets. We present a class of prior distributions parameterized by a rule set prototype that is stochastically modified to produce a task-specific rule set. We also describe a coordinate ascent algorithm that iteratively optimizes the task-specific rule sets and the prior distribution. Experiments using this algorithm show that transferring information from related tasks significantly reduces the amount of training data required to predict action effects in blocks-world domains.
BibTeX
@inproceedings{deshpande2007learning,
  author = {Ashwin Deshpande and Brian Milch and Luke S. Zettlemoyer and Leslie Pack Kaelbling},
  title = {{Learning Probabilistic Relational Dynamics for Multiple Tasks}},
  booktitle = {Probabilistic, Logical and Relational Learning - A Further Synthesis},
  year = {2007},
  url = {https://openreview.net/forum?id=vTzB3y4xwb2},
}
Figure