Introduction
Apprenticeship learning (AL) is a powerful paradigm where an agent learns to perform a task by observing trajectories provided by an expert. Traditionally, AL assumes that the expert and the learner operate within the same environment. However, in many real-world applications, expert demonstrations may come from multiple environments with varying dynamics—such as driving data collected across different cities, weather conditions, or vehicle types.
In these scenarios, a fundamental trade-off emerges. A policy optimized for a single environment may fail when dynamics shift, while a single "average" policy intended for all environments may lack the precision required for any specific one. This paper introduces the Cross Apprenticeship Learning (CAL) framework to address this challenge, providing a principled method to balance environment-specific expertise with global robustness.
The Cross Apprenticeship Learning (CAL) Framework
The CAL framework is designed to exploit expert data from multiple environments simultaneously. The core objective is to minimize the "worst-case discrepancy" between the learner's policy and the expert's policy across all available environments.
To manage the trade-off between specialization and generalization, the authors introduce a centrality measure ($\epsilon$). The optimization problem is structured such that each environment-specific policy ($\pi_i$) must remain within a certain distance of a shared, "cross-learned" central policy ($\pi_c$).
- High $\epsilon$ (Specialization): Allows individual policies to deviate significantly from the central policy, prioritizing high performance in their specific local environments.
- Low $\epsilon$ (Robustness): Forces individual policies to converge toward the central policy, ensuring more consistent performance across all environments.
Optimization and Convexification
Mathematically, the CAL framework builds upon the Linear Programming (LP) approach to apprenticeship learning, utilizing occupation measures—the discounted probability of an agent being in a specific state-action pair.
One significant hurdle is that the relationship between policies and occupation measures is bilinear, rendering the optimization problem non-convex. To overcome this, the authors employ McCormick envelopes. This technique creates a convex outer approximation of the non-convex feasibility set, transforming the problem into a computationally tractable form that can be solved using standard optimization tools. Furthermore, the authors provide mathematical guarantees for the framework, proving the upper semicontinuity of the set of optimizers, which ensures that small adjustments to the centrality parameter do not lead to erratic changes in the resulting policies.
Experimental Evaluation: Windy Gridworld
The framework was evaluated using a customized Windy Gridworld simulation consisting of four distinct $7 \times 10$ grids. Each gridworld featured a unique "wind vector" that altered transition probabilities by pushing the agent in specific directions. Expert trajectories were generated for each environment using the $\epsilon$-greedy SARSA algorithm, and the learner utilized 200 sample trajectories per environment to estimate expert occupation measures.
Key Results
The simulation results highlighted the impact of the centrality parameter on agent success:
- Specialization ($\epsilon = 1$): When the constraint was relaxed, policies were highly effective in their "home" environments (e.g., 199/200 successes) but performed poorly when transferred to others (dropping as low as 2/200 successes).
- Robustness ($\epsilon = 0.2$): Tightening the centrality constraint led to more uniform success rates. Individual policies achieved between 115 and 181 successes across all four gridworlds.
- The Approximation Advantage: Interestingly, $\epsilon = 0.2$ outperformed the perfect consensus case ($\epsilon = 0$). The authors attribute this to the McCormick relaxation; providing a small amount of "slack" in the constraints likely helped the solver find better global solutions within the approximate feasibility set.
Conclusion and Future Work
The CAL framework provides a robust mathematical foundation for learning from diverse expert demonstrations. By tuning a single parameter, researchers can navigate the tension between local specialization and global generalization.
While the results in gridworld environments are promising, several avenues for future research remain. These include scaling the framework to the massive state-action spaces typical of deep reinforcement learning, addressing the approximation gap inherent in McCormick relaxations, and developing fully distributed implementations where agents can collaborate without sharing raw local expert data.
Summary updated