If an agent needs to learn a versatile skill that works across many different scenarios, what exactly should it practice to get good as quickly as possible? This paper addresses a fundamental question in robotics and reinforcement learning by introducing a framework for the active learning of parameterized skills.
The Challenge of Parameterized Skills
In hierarchical reinforcement learning, we often want to develop reusable "skills" (or options). A parameterized skill is one that can adapt to a range of tasks—for example, a soccer robot learning a "kick" skill that must work for various distances and directions.
The primary challenge here is efficiency. Learning a policy for every possible variation of a task is computationally and temporally expensive. Most existing methods either sample tasks randomly or follow the target distribution. However, these "passive" strategies are highly inefficient:
- They waste time practicing tasks the agent has already mastered.
- They struggle with "hard" regions of the task space where small changes in the environment require vastly different control strategies (discontinuities in the policy manifold).
- They fail to account for how learning one task helps the agent generalize to its neighbors.
An Active Learning Framework
To solve this, the authors propose an active learning loop that treats skill learning as a Bayesian optimization problem, but with a unique twist tailored for reinforcement learning.
The framework relies on three core components:
- Performance Modeling: A Gaussian Process (GP) is used to model the expected return of a skill across the entire task space.
- EISP Acquisition Function: Standard Bayesian Optimization functions (like Expected Improvement) aim to find the maximum of a function. Here, the goal is to improve the entire function. The authors derive Expected Improvement in Skill Performance (EISP), which calculates the expected increase in the integral of performance over the whole task distribution if a specific task is practiced. They also provide a closed-form analytical gradient for EISP, allowing for efficient optimization.
- Spatiotemporal Kernel: Skill performance is inherently non-stationary—the agent gets better as it practices. Standard GP kernels assume the underlying function is static, meaning old "bad" performance data would perpetually pull down the current estimate. To fix this, the authors design a custom kernel that weights recent observations more heavily while maintaining spatial correlations, allowing the GP to accurately "track" the agent's improving competence.
Experimental Domain: The Simulated Catapult
The method is evaluated on a custom Simulated Catapult Domain. The agent must hit targets at various horizontal distances on a 2D mountainous terrain. The terrain is irregular and unknown to the agent. Because of the "mountains," a small change in target distance might require a completely different launch arc to avoid an obstacle, creating difficult-to-learn discontinuities.
Key Results and Insights
The authors compared EISP against random sampling, task-distribution sampling, and standard Bayesian Optimization metrics like Expected Improvement (EI) and Lower Confidence Bound (LCB).
- Superior Efficiency: As shown in the results, EISP reached higher levels of skill competence with significantly fewer training episodes than all other methods.
- Generalization is the Goal: The core insight is that an active learner shouldn't just look for where it is currently failing (like LCB) or where it is already succeeding (like EI). It should look for the task that provides the most "information gain" for the rest of the task distribution.
- Avoiding the "Hardness Trap": Purely curiosity-driven or error-based learners often get stuck trying to solve impossible tasks. Because EISP models the expected improvement, it can eventually realize when a region of the task space is too difficult to yield further gains and move on to more productive tasks. Qualitative results showed that EISP ignores "easy" tasks and focuses its efforts on the "ridges" and "cliffs" of the policy manifold where generalization is hardest.
Limitations and Future Work
While highly effective, the approach has a few limitations. First, EISP is "myopic"—it acts as a one-step look-ahead strategy, picking the best task for the next practice session rather than planning a long-term curriculum. Second, while the authors claim the method scales independently of task dimensionality, high-dimensional task spaces might still face the "curse of dimensionality" inherent to GPs. Finally, the model assumes constant observation noise across the task space (homoscedasticity), whereas real-world tasks often have varying noise levels. Future work could integrate this framework with Learning from Demonstration (LfD) to further bias the agent toward practicing relevant tasks identified by an expert.
Summary updated