Out-of-distribution generalization of internal models is correlated with reward

Khushdeep Singh Mann, Steffen Schneider, Alberto Silvio Chiappa, Jin Hwa Lee, Matthias Bethge, Alexander Mathis, Mackenzie W MathisOpenReview 2023 · OpenReview

Abstract

We investigate the behavior of reinforcement learning (RL) agents under morphological distribution shifts. Similar to recent robustness benchmarks in computer vision, we train algorithms on selected RL environments and test transfer performance on perturbed environments. We specifically test perturbations to popular RL agent’s morphologies by changing the length and mass of limbs, which in biological settings is a major challenge (eg, after injury or during growth). In this setup, called PyBullet-M, we compare the performance of policies obtained by reward-driven learning with self-supervised models of the observed state-action transitions. We find that out-of-distribution performance of self-supervised models is correlated to degradation in reward.
BibTeX
@misc{mann2023out,
  author = {Khushdeep Singh Mann and Steffen Schneider and Alberto Silvio Chiappa and Jin Hwa Lee and Matthias Bethge and Alexander Mathis and Mackenzie W Mathis},
  title = {{Out-of-distribution generalization of internal models is correlated with reward}},
  year = {2023},
  url = {https://openreview.net/forum?id=VMLRB9g08l},
  howpublished = {OpenReview},
}

Loading related content...

Figure