Abstract
A popular approach to apprenticeship learning (AL) is to formulate it as an inverse reinforcement learning (IRL) problem. The MaxEnt-IRL algorithm successfully integrates the maximum entropy principle into IRL and unlike its predecessors, it resolves the ambiguity arising from the fact that a possibly large number of policies could match the expert's behavior. In this paper, we study an AL setting in which in addition to the expert's trajectories, a number of unsupervised trajectories is available. We introduce MESSI, a novel algorithm that combines MaxEnt-IRL with principles coming from semi-supervised learning. In particular, MESSI integrates the unsupervised data into the MaxEnt-IRL framework using a pairwise penalty on trajectories. Empirical results in a highway driving and grid-world problems indicate that MESSI is able to take advantage of the unsupervised trajectories and improve the performance of MaxEnt-IRL.
Introduction
Apprenticeship Learning (AL) aims to train autonomous agents by mimicking expert behavior. A common strategy is Inverse Reinforcement Learning (IRL), where the agent infers the underlying reward function that the expert is likely maximizing. While effective, standard IRL faces significant challenges: expert demonstrations are often scarce and expensive to obtain, and multiple reward functions can often explain the same observed behavior.
To address these hurdles, we introduce MESSI (Maximum Entropy Semi-Supervised Inverse Reinforcement Learning). MESSI leverages the abundance of "unsupervised" or unlabeled trajectory data—such as recordings of non-expert drivers—to supplement limited expert demonstrations. By incorporating semi-supervised learning principles, MESSI reduces ambiguity and improves the robustness of the learned policy.
The MESSI Framework
MESSI builds upon the Maximum Entropy IRL (MaxEnt-IRL) framework, which selects the highest entropy distribution of trajectories that matches the expert's feature counts. The core innovation of MESSI is the addition of a Pairwise Penalty term to the optimization objective.
Manifold Assumption and Smoothness
MESSI operates on the "manifold assumption": physically similar trajectories should receive similar rewards. We define a similarity function (e.g., an RBF kernel) based on the features visited during a trajectory. The pairwise penalty $R(\theta|\Sigma)$ penalizes reward functions that assign vastly different values to similar trajectories. The final objective function balances the likelihood of expert data with this smoothness constraint:
$$\theta^* = \text{argmax}_\theta (L(\theta|\Sigma^*) - \lambda R(\theta|\Sigma))$$
This approach allows the agent to "fill in the gaps" of the state space that the expert may not have visited, learning the relative value of those states from unlabeled data.
Empirical Evaluation
We evaluated MESSI in two environments: a complex Highway Driving simulator and a standard Grid-World navigation task. Unsupervised data was generated from a mix of near-expert, random, and noisy policies.
Performance and Convergence
Experimental results demonstrate that MESSI significantly outperforms standard MaxEnt-IRL, particularly when expert data is extremely limited (e.g., a single trajectory). As shown in the learning curves, MESSI variants achieve higher rewards more quickly than the baseline.
Comparison with EM Baselines
We also compared MESSI against a semi-supervised baseline using the Expectation-Maximization (EM) algorithm. MESSI proved to be more stable and achieved superior final rewards. EM-based methods often suffered from instability, as early estimation errors tended to be amplified over successive iterations.
Key Insights
- Robustness to Noise: MESSI remains effective even when the unlabeled data is "noisy" (e.g., only 15% expert-like). It only degrades when the unlabeled data is completely irrelevant to the task.
- Information Completion: By observing non-expert trajectories (such as off-road accidents), the agent learns the negative weights of features the expert successfully avoids, leading to a more robust reward function.
- Numerical Stability: To ensure practical performance, we implemented feature discounting and weight constraints to prevent gradient explosion during optimization.
Conclusion
MESSI provides a powerful framework for semi-supervised apprenticeship learning. By integrating a pairwise penalty into the MaxEnt-IRL objective, it effectively utilizes unlabeled data to overcome expert data scarcity. Future work will focus on extending this manifold-based approach to highly stochastic environments using Causal Entropy IRL.
Summary updated