Cross Apprenticeship Learning Framework: Properties and Solution Approaches

Ashwin Aravind, Debasish Chatterjee, Ashish Cherukuri

TL;DR

Standard apprenticeship learning (AL) focuses on mimicking an expert in a single environment. This paper introduces the Cross Apprenticeship Learning (CAL) framework, which allows an agent to learn from multiple experts across different environments with varying dynamics. By tuning a single "centrality" parameter, the framework allows researchers to strike a principled balance between environment-specific expertise and robust performance across all scenarios.

What Problem Does This Paper Address?

In traditional apprenticeship learning, an agent observes an expert’s trajectories to recover a policy that performs a specific task. However, real-world dynamics are rarely static. An expert might demonstrate a task (like driving) in several different environments (different cities, weather conditions, or vehicle types).

The challenge lies in the trade-off:

  1. Specific Performance: If you train strictly on one environment, the policy might fail when the dynamics change slightly.
  2. Generic Performance: If you try to find one "average" policy for everything, it might be mediocre in every specific environment.

The authors tackle the problem of how to exploit expert data from multiple environments to learn policies that are both locally effective and globally robust.

Methods

The authors build upon the Linear Programming (LP) approach to apprenticeship learning, which uses occupation measures—essentially the discounted probability of being in a certain state-action pair.

The core of the CAL framework is a multi-objective optimization problem:

  • Objective: Minimize the "worst-case discrepancy" between the learner's policy and the expert's policy across all environments.
  • Constraint: Each environment-specific policy ($\pi_i$) must remain within a certain distance ($\epsilon$) of a shared "cross-learned" policy ($\pi_c$).
  • The Tuning Knob: The parameter $\epsilon$, termed the centrality measure, controls the trade-off. A large $\epsilon$ allows policies to specialize (high specific performance), while a small $\epsilon$ forces them to converge (high generic performance).

Technically, the problem is non-convex because the relationship between policies and occupation measures is bilinear. To solve this, the authors employ McCormick envelopes, a technique used to create a convex outer approximation of the non-convex feasibility set, making the problem computationally tractable.

Datasets

The framework was evaluated using a customized Windy Gridworld simulation:

  • Environment: Four distinct $7 \times 10$ gridworlds.
  • Dynamics: Each gridworld has a different "wind vector" that pushes the agent in specific directions, changing the transition probabilities.
  • Expert Data: Expert trajectories were generated for each gridworld using the $\epsilon$-greedy SARSA algorithm. The learner used 200 sample trajectories per environment to estimate the expert's occupation measures.

Key Insights and Ideas

  • The Power of Centrality: The authors prove that the performance of the policies is mathematically tied to the centrality parameter. They demonstrate that as $\epsilon$ decreases, the individual policies naturally inherit the "generic" strengths of the group.
  • Mathematical Guarantees: The paper provides a proof for the upper semicontinuity of the set of optimizers. In plain English: small changes in your tuning parameter won't cause wild, unpredictable jumps in the resulting policies.
  • Convexification Strategy: By transforming the policy-search problem into an occupation-measure problem and then applying McCormick relaxations, the authors provide a roadmap for solving complex, non-convex reinforcement learning constraints using standard optimization tools.

Key Results

The simulation results (Table 1) clearly demonstrate the framework's flexibility:

  • Specialization ($\epsilon = 1$): Individual policies were nearly perfect in their "home" environments (e.g., 199/200 successes) but failed miserably in others (sometimes as low as 2/200 successes).
  • Robustness ($\epsilon = 0.2$): By tightening the centrality constraint, the success rate became much more uniform. Individual policies achieved between 115 and 181 successes across all four gridworlds.
  • The "Aha" Moment: Interestingly, $\epsilon = 0.2$ actually outperformed the perfect consensus case ($\epsilon = 0$) across all environments. The authors suggest this is because the McCormick method finds an approximate optimizer, and a little bit of "slack" in the constraints actually helps the solver find better global solutions.

Limitations and Future Work

  • Scalability: While effective for gridworlds, the authors acknowledge that the current approach may face challenges when scaling to massive state-action spaces common in deep RL.
  • Approximation Gap: The McCormick relaxation provides an outer approximation. This means the resulting policies might not perfectly satisfy the original distance constraints, requiring a "projection" step to ensure feasibility.
  • Distributed Implementation: While the paper discusses the potential for distributed computation (where agents don't have to share their local expert data), a full implementation of distributed bilinear CAL remains a task for future research.

Summary updated

Abstract

Apprenticeship learning is a framework in which an agent learns a policy to perform a given task in an environment using example trajectories provided by an expert. In the real world, one might have access to expert trajectories in different environments where system dynamics is different while the learning task is the same. For such scenarios, two types of learning objectives can be defined. One where the learned policy performs very well in one specific environment and another when it performs well across all environments. To balance these two objectives in a principled way, our work presents the cross apprenticeship learning (CAL) framework. This consists of an optimization problem where an optimal policy for each environment is sought while ensuring that all policies remain close to each other. This nearness is facilitated by one tuning parameter in the optimization problem. We derive properties of the optimizers of the problem as the tuning parameter varies. We identify conditions under which an agent prefers using the policy obtained from CAL over the traditional apprenticeship learning. Since the CAL problem is nonconvex, we provide a convex outer approximation. Finally, we demonstrate the attributes of our framework in the context of a navigation task in a windy gridworld environment.
BibTeX
@article{aravind2023cross,
  author = {Ashwin Aravind and Debasish Chatterjee and Ashish Cherukuri},
  title = {{Cross Apprenticeship Learning Framework: Properties and Solution Approaches}},
  journal = {IEEE Open Journal of Control Systems},
  year = {2023},
  doi = {10.1109/ojcsys.2023.3235248},
}
Figure