Learning Environmental Calibration Actions for Policy Self-Evolution

Chao Zhang, Yang Yu, Zhi-Hua ZhouIJCAI 2018 · OpenReview

TL;DR

This paper introduces POSEC, a novel approach for rapidly adapting reinforcement learning policies to new physical environments. POSEC learns to perform a few "calibration actions" in a new environment to quickly understand its dynamics, allowing it to reuse pre-trained policies effectively with minimal new data. This significantly reduces the sample complexity of deploying RL in real-world scenarios.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn by trial and error, which is incredibly sample-intensive. While simulators can speed up training, policies trained in simulators often perform poorly in the real world due to simulation inaccuracies. Deploying RL in physical systems (like robots) is expensive and time-consuming, making it impractical to retrain policies from scratch for every new robot or slightly altered environment. The core challenge is how to efficiently adapt existing, pre-trained policies to a new, real-world environment without requiring millions of costly real-world interactions.

Methods

The paper proposes the POSEC (Policy Self-Evolution by Calibration) approach. POSEC operates in two stages:

  1. Offline Training:
    • Base Policies: A large set of diverse base policies are trained in a parameterized simulator across many simulated environments. These policies are trained using standard RL algorithms like TRPO.
  2. Combination Weights: For a set of simulated environments, the best linear combination weights for the base policies are learned to maximize performance.
  3. Calibration Actions & Regression Model: This is the core innovation. POSEC learns a set of "calibration actions" and a regression model. The calibration actions, when executed in an environment, produce observations. These observations are then used as features to predict the optimal combination weights for the base policies via the learned regression model. This stage uses derivative-free optimization to find the best calibration actions and regression coefficients.
  4. Online Self-Evolution (in the target environment):
    • When deployed in a new environment, the agent executes the learned calibration actions.
  5. The observations from these actions are fed into the learned regression model to predict the optimal combination weights.
  6. A self-evolved policy is formed by linearly combining the pre-trained base policies using these predicted weights. This policy is then ready to be executed.

Datasets

The paper uses three robotic arm manipulation tasks from the OpenAI Gym environment: Pusher, Striker, and Thrower. For training, they generate 100 different environments by randomly varying robotic arm link lengths. For evaluating the combination weights and calibration actions, they use separate sets of 20 environments each. These environments are generated by randomizing simulator parameters, creating variations of the base tasks.

Key Insights and Ideas

  • Calibration Actions as Environment Probes: The key insight is that a few carefully chosen "calibration actions" can act as highly informative probes to quickly characterize an unknown environment's dynamics. This is more effective than random actions.
  • Reducing Policy Learning to Environment Identification: POSEC reframes the problem of learning a new policy in a new environment from a full RL training task to a much simpler "environment identification" task (i.e., figuring out which combination of base policies is best).
  • Leveraging Pre-trained Policies: The approach effectively leverages a library of pre-trained base policies, making the adaptation process much faster than learning from scratch.
  • Sample Efficiency is Paramount: The paper strongly emphasizes the need for sample efficiency in real-world RL, where data collection is expensive. POSEC's ability to achieve good performance with very few samples is its main selling point.

Key Results

  • POSEC, using only 1, 5, or 10 calibration actions, achieved significantly better performance than using a single random action for environment probing.
  • With just 1 calibration action (and thus 1 sample from the new environment), POSEC outperformed training a TRPO policy from scratch using millions of samples on the Pusher and Striker tasks. On the Thrower task, 5 calibration actions were needed to surpass the millions-of-samples baseline.
  • POSEC's self-evolved policies served as excellent initial policies for further refinement training using standard RL algorithms, consistently leading to better final performance compared to other initialization methods.

Limitations and Future Work

  • Linear Combination: The current work focuses on linear combinations of base policies. The authors suggest that exploring non-linear combinations could be beneficial, though it might require more samples.
  • Simulator Assumption: The offline training relies on a parameterized simulator that can generate diverse environments. The effectiveness in the real world depends on how well the simulated environments cover the variations encountered in reality.
  • Generalization to Different State/Action Spaces: The current POSEC assumes the new environment has the same state and action spaces as the training environments. Future work aims to extend policy self-evolution to scenarios with different state and/or action spaces.

Summary updated

Abstract

Reinforcement learning in physical world is often expensive. Simulators are commonly employed to train policies. Due to the simulation error, trained-in-simulator policies are hard to be directly deployed in physical world. Therefore, how to efficiently reuse these policies to the real environment is a key issue. To address this issue, this paper presents a policy self-evolution process: in the target environment, the agent firstly executes a few calibration actions to perceive the environment, and then reuses the previous policies according to the observation of the environment. In this way, the mission of policy learning in the target environment is reduced to the task of environment identification through executing the calibration actions, which needs much less samples than learning a policy from scratch. We propose the POSEC (POlicy Self-Evolution by Calibration) approach, which learns the most informative calibration actions for policy self-evolution. Taking three robotic arm controlling tasks as the test beds, we show that the proposed method can learn a fine policy for a new arm with only a few (e.g. five) samples of the target environment.
BibTeX
@inproceedings{zhang2018learning,
  author = {Chao Zhang and Yang Yu and Zhi-Hua Zhou},
  title = {{Learning Environmental Calibration Actions for Policy Self-Evolution}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=S1Erv7z_bH},
}
Figure