TL;DR
Standard apprenticeship learning (AL) focuses on mimicking an expert in a single environment. This paper introduces the Cross Apprenticeship Learning (CAL) framework, which allows an agent to learn from multiple experts across different environments with varying dynamics. By tuning a single "centrality" parameter, the framework allows researchers to strike a principled balance between environment-specific expertise and robust performance across all scenarios.
What Problem Does This Paper Address?
In traditional apprenticeship learning, an agent observes an expert’s trajectories to recover a policy that performs a specific task. However, real-world dynamics are rarely static. An expert might demonstrate a task (like driving) in several different environments (different cities, weather conditions, or vehicle types).
The challenge lies in the trade-off:
- Specific Performance: If you train strictly on one environment, the policy might fail when the dynamics change slightly.
- Generic Performance: If you try to find one "average" policy for everything, it might be mediocre in every specific environment.
The authors tackle the problem of how to exploit expert data from multiple environments to learn policies that are both locally effective and globally robust.
Methods
The authors build upon the Linear Programming (LP) approach to apprenticeship learning, which uses occupation measures—essentially the discounted probability of being in a certain state-action pair.
The core of the CAL framework is a multi-objective optimization problem:
- Objective: Minimize the "worst-case discrepancy" between the learner's policy and the expert's policy across all environments.
- Constraint: Each environment-specific policy ($\pi_i$) must remain within a certain distance ($\epsilon$) of a shared "cross-learned" policy ($\pi_c$).
- The Tuning Knob: The parameter $\epsilon$, termed the centrality measure, controls the trade-off. A large $\epsilon$ allows policies to specialize (high specific performance), while a small $\epsilon$ forces them to converge (high generic performance).
Technically, the problem is non-convex because the relationship between policies and occupation measures is bilinear. To solve this, the authors employ McCormick envelopes, a technique used to create a convex outer approximation of the non-convex feasibility set, making the problem computationally tractable.
Datasets
The framework was evaluated using a customized Windy Gridworld simulation:
- Environment: Four distinct $7 \times 10$ gridworlds.
- Dynamics: Each gridworld has a different "wind vector" that pushes the agent in specific directions, changing the transition probabilities.
- Expert Data: Expert trajectories were generated for each gridworld using the $\epsilon$-greedy SARSA algorithm. The learner used 200 sample trajectories per environment to estimate the expert's occupation measures.
Key Insights and Ideas
- The Power of Centrality: The authors prove that the performance of the policies is mathematically tied to the centrality parameter. They demonstrate that as $\epsilon$ decreases, the individual policies naturally inherit the "generic" strengths of the group.
- Mathematical Guarantees: The paper provides a proof for the upper semicontinuity of the set of optimizers. In plain English: small changes in your tuning parameter won't cause wild, unpredictable jumps in the resulting policies.
- Convexification Strategy: By transforming the policy-search problem into an occupation-measure problem and then applying McCormick relaxations, the authors provide a roadmap for solving complex, non-convex reinforcement learning constraints using standard optimization tools.
Key Results
The simulation results (Table 1) clearly demonstrate the framework's flexibility:
- Specialization ($\epsilon = 1$): Individual policies were nearly perfect in their "home" environments (e.g., 199/200 successes) but failed miserably in others (sometimes as low as 2/200 successes).
- Robustness ($\epsilon = 0.2$): By tightening the centrality constraint, the success rate became much more uniform. Individual policies achieved between 115 and 181 successes across all four gridworlds.
- The "Aha" Moment: Interestingly, $\epsilon = 0.2$ actually outperformed the perfect consensus case ($\epsilon = 0$) across all environments. The authors suggest this is because the McCormick method finds an approximate optimizer, and a little bit of "slack" in the constraints actually helps the solver find better global solutions.
Limitations and Future Work
- Scalability: While effective for gridworlds, the authors acknowledge that the current approach may face challenges when scaling to massive state-action spaces common in deep RL.
- Approximation Gap: The McCormick relaxation provides an outer approximation. This means the resulting policies might not perfectly satisfy the original distance constraints, requiring a "projection" step to ensure feasibility.
- Distributed Implementation: While the paper discusses the potential for distributed computation (where agents don't have to share their local expert data), a full implementation of distributed bilinear CAL remains a task for future research.
Summary updated