Smooth Imitation Learning via Smooth Costs and Smooth Policies

Sapana Chaudhary, Balaraman Ravindran

Abstract

Imitation learning (IL) is a popular approach in the continuous control setting as among other reasons it circumvents the problems of reward mis-specification and exploration in reinforcement learning (RL). In IL from demonstrations, an important challenge is to obtain agent policies that are smooth with respect to the inputs. Learning through imitation a policy that is smooth as a function of a large state-action ($s$-$a$) space (typical of high dimensional continuous control environments) can be challenging. We take a first step towards tackling this issue by using smoothness inducing regularizers on \textit{both} the policy and the cost models of adversarial imitation learning. Our regularizers work by ensuring that the cost function changes in a controlled manner as a function of $s$-$a$ space; and the agent policy is well behaved with respect to the state space. We call our new smooth IL algorithm \textit{Smooth Policy and Cost Imitation Learning} (SPaCIL, pronounced 'Special'). We introduce a novel metric to quantify the smoothness of the learned policies. We demonstrate SPaCIL's superior performance on continuous control tasks from MuJoCo. The algorithm not just outperforms the state-of-the-art IL algorithm on our proposed smoothness metric, but, enjoys added benefits of faster learning and substantially higher average return.
BibTeX
@misc{chaudhary2021smooth,
  author = {Sapana Chaudhary and Balaraman Ravindran},
  title = {{Smooth Imitation Learning via Smooth Costs and Smooth Policies}},
  year = {2021},
  eprint = {2111.02354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.02354},
}

Loading related content...

Figure