Cross apprenticeship learning framework: Properties and solution approaches

Ashwin Aravind, Debasish Chatterjee, Ashish Cherukuri

Introduction

Apprenticeship learning (AL) is a powerful paradigm where an agent learns to perform a task by observing trajectories provided by an expert. Traditionally, AL assumes that the expert and the learner operate within the same environment. However, in many real-world applications, expert demonstrations may come from multiple environments with varying dynamics—such as driving data collected across different cities, weather conditions, or vehicle types.

In these scenarios, a fundamental trade-off emerges. A policy optimized for a single environment may fail when dynamics shift, while a single "average" policy intended for all environments may lack the precision required for any specific one. This paper introduces the Cross Apprenticeship Learning (CAL) framework to address this challenge, providing a principled method to balance environment-specific expertise with global robustness.

The Cross Apprenticeship Learning (CAL) Framework

The CAL framework is designed to exploit expert data from multiple environments simultaneously. The core objective is to minimize the "worst-case discrepancy" between the learner's policy and the expert's policy across all available environments.

To manage the trade-off between specialization and generalization, the authors introduce a centrality measure ($\epsilon$). The optimization problem is structured such that each environment-specific policy ($\pi_i$) must remain within a certain distance of a shared, "cross-learned" central policy ($\pi_c$).

  • High $\epsilon$ (Specialization): Allows individual policies to deviate significantly from the central policy, prioritizing high performance in their specific local environments.
  • Low $\epsilon$ (Robustness): Forces individual policies to converge toward the central policy, ensuring more consistent performance across all environments.

Optimization and Convexification

Mathematically, the CAL framework builds upon the Linear Programming (LP) approach to apprenticeship learning, utilizing occupation measures—the discounted probability of an agent being in a specific state-action pair.

One significant hurdle is that the relationship between policies and occupation measures is bilinear, rendering the optimization problem non-convex. To overcome this, the authors employ McCormick envelopes. This technique creates a convex outer approximation of the non-convex feasibility set, transforming the problem into a computationally tractable form that can be solved using standard optimization tools. Furthermore, the authors provide mathematical guarantees for the framework, proving the upper semicontinuity of the set of optimizers, which ensures that small adjustments to the centrality parameter do not lead to erratic changes in the resulting policies.

Experimental Evaluation: Windy Gridworld

The framework was evaluated using a customized Windy Gridworld simulation consisting of four distinct $7 \times 10$ grids. Each gridworld featured a unique "wind vector" that altered transition probabilities by pushing the agent in specific directions. Expert trajectories were generated for each environment using the $\epsilon$-greedy SARSA algorithm, and the learner utilized 200 sample trajectories per environment to estimate expert occupation measures.

Key Results

The simulation results highlighted the impact of the centrality parameter on agent success:

  1. Specialization ($\epsilon = 1$): When the constraint was relaxed, policies were highly effective in their "home" environments (e.g., 199/200 successes) but performed poorly when transferred to others (dropping as low as 2/200 successes).
  2. Robustness ($\epsilon = 0.2$): Tightening the centrality constraint led to more uniform success rates. Individual policies achieved between 115 and 181 successes across all four gridworlds.
  3. The Approximation Advantage: Interestingly, $\epsilon = 0.2$ outperformed the perfect consensus case ($\epsilon = 0$). The authors attribute this to the McCormick relaxation; providing a small amount of "slack" in the constraints likely helped the solver find better global solutions within the approximate feasibility set.

Conclusion and Future Work

The CAL framework provides a robust mathematical foundation for learning from diverse expert demonstrations. By tuning a single parameter, researchers can navigate the tension between local specialization and global generalization.

While the results in gridworld environments are promising, several avenues for future research remain. These include scaling the framework to the massive state-action spaces typical of deep reinforcement learning, addressing the approximation gap inherent in McCormick relaxations, and developing fully distributed implementations where agents can collaborate without sharing raw local expert data.

Summary updated

Abstract

Apprenticeship learning is a framework in which an agent learns a policy to perform a given task in an environment using example trajectories provided by an expert. In the real world, one might have access to expert trajectories in different environments where the system dynamics is different while the learning task is the same. For such scenarios, two types of learning objectives can be defined. One where the learned policy performs very well in one specific environment and another when it performs well across all environments. To balance these two objectives in a principled way, our work presents the cross apprenticeship learning (CAL) framework. This consists of an optimization problem where an optimal policy for each environment is sought while ensuring that all policies remain close to each other. This nearness is facilitated by one tuning parameter in the optimization problem. We derive properties of the optimizers of the problem as the tuning parameter varies. Since the problem is nonconvex, we provide a convex outer approximation. Finally, we demonstrate the attributes of our framework in the context of a navigation task in a windy gridworld environment.
BibTeX
@article{aravind2023cross,
  author = {Ashwin Aravind and Debasish Chatterjee and Ashish Cherukuri},
  title = {{Cross apprenticeship learning framework: Properties and solution approaches}},
  journal = {IEEE Open Journal of Control Systems},
  year = {2023},
  doi = {10.1109/ojcsys.2023.3235248},
}
Figure