Authors: Luke S. Zettlemoyer and Leslie Pack Kaelbling (MIT CSAIL)
Introduction
In the field of autonomous agents, a significant hurdle is the ability to generalize knowledge across different environments. While a robot might master a specific task—such as opening a cabinet in a laboratory setting—it often struggles when relocated to a new environment where physical properties differ slightly. Traditional reinforcement learning and planning algorithms frequently require the agent to relearn dynamics from scratch in every new scenario. This paper presents a hierarchical Bayesian approach designed to facilitate transfer learning by capturing and sharing the underlying structure of action dynamics across multiple related tasks.
The Challenge of Structural Transfer
Most existing transfer learning methods focus on adjusting continuous, real-valued parameters. However, action effects in complex environments are often best modeled using probabilistic relational planning rules. These rules are discrete logical structures, such as: "If an agent pulls a handle, the cabinet opens with probability p."
Transferring these discrete structures is technically challenging because they do not inhabit a continuous space where standard statistical methods like Hierarchical Bayesian Models (HBMs) are typically applied. The goal is to learn a global prior that can guide the discovery of these logical rules in new, unseen tasks, thereby reducing the amount of data required for an agent to become proficient.
Hierarchical Bayesian Model
The core contribution of this work is an HBM tailored for discrete rule sets. The model assumes that while each task $k$ has its own specific rule set $R_k$, these sets are all derived from a common global "prototype" $G$.
As shown in Figure 1, the model structure is as follows:
- Rule Set Prototype ($G$): This acts as a global prior. It contains "rule prototypes" that define distributions (via Dirichlet parameters) over possible action outcomes.
- Task-Specific Rule Sets ($R_k$): These are modeled as stochastic modifications of the prototype $G$. The model accounts for the possibility that a specific task might add new rules, delete existing ones, or modify the preconditions and outcomes of the prototype rules.
- Observations ($x_{kn}$): These represent the actual data points—the observed effects of actions—collected within each specific task environment.
By using a relational representation (first-order logic), the rules describe interactions between general classes of objects (e.g., "blocks" or "containers") rather than specific instances. This allows the learned dynamics to generalize across different numbers and types of objects within a domain.
Learning via Coordinate Ascent
To estimate the optimal prototype and task-specific rules, the authors employ a coordinate ascent algorithm. This iterative process alternates between two main steps:
- Task Optimization: Given the current global prototype $G$, the algorithm optimizes each task-specific rule set $R_k$ to best fit the local observations.
- Prototype Optimization: The global prototype $G$ is then updated to be the most likely common ancestor for the current set of task-specific rules $\{R_1, ..., R_K\}$.
This approach allows the prototype to act as a powerful search bias. Instead of searching the entire space of possible logical formulas, the algorithm prioritizes rules that are structurally similar to the prototype, significantly narrowing the search space and improving data efficiency.
Experimental Evaluation
The model was tested on several variations of the "Blocks World" domain, a standard benchmark in AI planning. These domains were designed to test different types of variation:
- Gripper Size: Tasks varied based on block and gripper dimensions; only matching sizes allowed successful manipulation.
- Slippery Gripper: Tasks varied in the probability of dropping a block, influenced by a hidden "wet" state.
- Slippery Gripper with Size: A complex domain combining both structural (size) and parametric (probability) variations.
- Random Domain: A control domain where tasks were entirely unrelated, used to evaluate the risk of "negative transfer."
Key Results and Insights
The experiments demonstrated that the hierarchical approach provides substantial benefits in terms of data efficiency. In the Gripper Size domain, the transfer learner achieved near-peak accuracy with approximately 40 examples, whereas a baseline model without transfer required over 100 examples to reach the same performance level.
Furthermore, the model proved robust against negative transfer. In the Random Domain, where no meaningful commonality existed between tasks, the hierarchical model did not perform worse than the baseline. It simply learned a non-specific prototype, effectively "turning off" the transfer mechanism when it was not beneficial.
Conclusion and Future Work
This research demonstrates that hierarchical Bayesian modeling can be successfully extended to discrete, relational structures for transfer learning. By learning a rule set prototype, agents can rapidly adapt to new environments by identifying which variations of a general rule apply.
Future work aims to address the computational complexity of searching discrete rule spaces and to scale the approach to more complex domains with a wider variety of actions. Additionally, moving beyond synthetic benchmarks to real-world robotic sensor data remains a key challenge for the practical application of these relational dynamics.
Summary updated