TL;DR
This paper introduces an incremental model-based reinforcement learning (MBRL) approach that learns planning actions over episodes without relying on past training data. It proposes "reliability" as an intrinsic motivation to guide learning and a novel mechanism to learn from failures, preventing repeated mistakes and improving efficiency.
What Problem Does This Paper Address?
Developing effective plans in complex, real-world domains often requires accurate models of how actions affect the environment. However, hand-coding these models is a significant challenge (the "knowledge engineering problem"). While learning models from data is an alternative, traditional methods struggle with non-stationary environments where the underlying dynamics change over time. Furthermore, learning from limited data within a single episode can be insufficient. This paper addresses the need for an MBRL approach that can learn and refine action models incrementally within an episode, adapt to changing environments, and efficiently learn from limited experiences, even when failures occur.
Methods
The core of the proposed method is the Incremental Learning Model (ILM). Key components include:
- Incremental Learning: ILM learns and refines action models episode by episode. Crucially, it does not use data from previous episodes, making it suitable for non-stationary environments. It leverages "prior action models" which can be hand-coded incomplete models or learned models from previous episodes.
- Reliability as Intrinsic Motivation: A novel concept, "reliability," is introduced. It's an empirical estimate of an action model's learning progress. Reliability guides the search for better models and influences exploration in reinforcement learning, favoring less reliable actions.
- Learning from Failure: ILM records failed action executions (due to unsatisfied preconditions) and uses this information to build a "tabu" list. This list prevents the agent from attempting actions in states that are known to lead to failure, thereby reducing inefficient exploration and improving learning.
- Rule-Based Representation: Action models are represented using relational rules, allowing for generalization across states, unlike propositional rules.
- Model Refinement: The paper extends an existing relational rule learner. The learning process is guided by a score function that incorporates a "deviation penalty" to keep learned models grounded around prior action models. Learning is delayed until certain criteria are met, preventing premature generalization from limited data.
Datasets
The paper evaluates ILM on three planning domains:
- Tireworld: A domain where cars can get flat tires, leading to potential dead-ends if no spare is available.
- Exploding Blocksworld: A domain where blocks can detonate when placed, potentially destroying parts of the environment.
- Logistics: A domain involving moving parcels with trucks and airplanes.
These domains are standard benchmarks from the International Probabilistic Planning Competition. The experiments involve planning problems of increasing scale within each domain.
Key Insights and Ideas
- Reliability is Key: Using "reliability" as an intrinsic motivation is a clever way to guide exploration and model learning. It allows the agent to prioritize actions that are better understood, leading to more efficient learning.
- Learning from Failure is Powerful: The "tabu" list mechanism for learning from failures is a significant contribution. It directly addresses a common issue in RL where failed attempts can be uninformative or even detrimental to learning. By preventing repeated failures, the agent conserves resources and learns more effectively.
- No Past Data, No Problem: The ability to learn effectively without relying on historical data from previous episodes is crucial for non-stationary environments. This makes the approach more robust to dynamic real-world scenarios.
- Prior Knowledge Matters: The framework's ability to incorporate prior action models (even incomplete ones) provides a strong starting point, accelerating learning, especially in complex domains.
Key Results
- Reduced Variational Distance: ILM consistently showed a decrease in the variational distance between its learned models and the true models across episodes in all three domains, indicating successful model learning.
- Improved Goal-Directedness: ILM achieved a higher average number of successful trials (reaching the goal state) compared to its variants (ILM-R, ILM-T) and R-MAX, especially in more complex scenarios.
- Learning from Failure Advantage: ILM-T (which did not learn from failure) and R-MAX performed significantly worse, particularly in domains with a higher probability of failure, highlighting the importance of the failure-learning mechanism.
- ILM-R's Struggle: ILM-R, which did not use reliability, performed worse than ILM in some cases, suggesting that weighting relational counts by reliability is beneficial.
Limitations and Future Work
- Complex Domains Require More Data: The authors acknowledge that for highly complex domains, more training data is still needed to learn accurate action models.
- Handling Non-Stationarity: While ILM doesn't use past episode data, the paper suggests future work on learning distributions from current training data only to better handle non-stationarity.
- Computational Efficiency: For the second issue of increased computation time with more data, future work could involve maintaining a fixed-size training dataset by replacing older experiences while maximizing variability.
- General Planner Integration: The paper notes that ILM can integrate with any planner that accepts PPDDL, suggesting potential for broader applicability.
Summary updated