Soft Action Priors: Towards Robust Policy Transfer

Matheus Centa, Philippe PreuxAAAI 2023 · DOI

TL;DR

This paper introduces "soft action priors" – imperfect but informative guidance for reinforcement learning (RL) agents – to improve sample efficiency and robustness. They propose adaptive algorithms that dynamically adjust the influence of these priors based on their estimated usefulness in different states, outperforming existing methods, especially when dealing with suboptimal guidance.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often require vast amounts of data to learn effective policies, a problem known as sample inefficiency. While prior knowledge can accelerate learning, traditional methods often assume this knowledge comes from a perfect "expert" or in the form of complete trajectories. However, in many real-world scenarios, such as healthcare or robotics, we might only have access to imperfect or "soft" guidance that isn't necessarily optimal but still offers valuable hints about good actions. Existing policy distillation techniques struggle to effectively leverage these suboptimal priors, leading to brittle performance. This paper aims to bridge this gap by developing methods that can robustly exploit these soft action priors, even when they contain errors or are only partially correct.

Methods

The core contribution is the development of adaptive policy distillation algorithms. These methods build upon the "Reinforcement Learning as Inference" framework, which naturally incorporates action priors. The key innovation is the introduction of prior weights, denoted by $\omega_{\pi_0}(s)$, which are learned parameters that dynamically adjust the influence of the action prior $\pi_0$ at each state $s$.

Specifically, the authors propose:

  • Prior Trick: A mathematical technique to derive reward bonuses that effectively incorporate non-uniform action priors, linking to existing entropy-regularized policy distillation methods.
  • Adaptive Reward Bonuses: Instead of applying a fixed reward bonus derived from the prior, their adaptive methods scale this bonus using the learned prior weights. This allows the agent to rely more on the prior in states where it's deemed useful and less where it might be misleading.
  • Structural Degradation: To realistically simulate imperfect priors, they introduce a "structural degradation" setting. This involves selectively replacing the expert prior with an adversarial policy in specific states, mimicking situations where a prior might be consistently wrong in certain contexts, rather than just randomly noisy.
  • Adaptive E2R (AE2R): An adaptive variant of the E2R algorithm that uses these prior weights to modulate the reward bonus and regularization loss. The prior weights are learned by minimizing a loss function that relates to the agent's value function estimates and TD errors.

Datasets

The paper uses two main experimental setups:

  • Tabular Grid Worlds: A benchmark environment consisting of 20x20 Grid Worlds with varying complexities and goal states. This setup, adapted from previous work, allows for statistically relevant results by averaging over one thousand different MDPs.
  • Continuous Control Benchmarks (MuJoCo): Standard robotics simulation environments from OpenAI Gym, including Hopper-v3, Walker2d-v3, HalfCheetah-v3, Ant-v3, and Humanoid-v3. These are used to evaluate the methods in more realistic, high-dimensional, and complex deep RL settings.

Key Insights and Ideas

  • Suboptimal Priors are Common and Useful: The paper argues that real-world guidance is rarely perfect. However, even imperfect guidance can significantly speed up learning if leveraged correctly.
  • State-Dependent Prior Influence is Crucial: The effectiveness of a prior can vary drastically depending on the current state. An adaptive approach that learns when to trust the prior is more robust than a static one.
  • Structural Degradation is More Realistic: Randomly adding noise to an expert prior is less representative of real-world imperfect guidance than selectively degrading the prior in specific states or contexts.
  • Adaptive Methods Enhance Robustness: The proposed adaptive algorithms demonstrate a significant improvement in robustness, particularly when dealing with structurally degraded or imperfect priors, outperforming non-adaptive baselines.

Key Results

  • Tabular Experiments: Adaptive methods (AER and AE2R) consistently outperformed existing algorithms (ER and E2R) when learning from imperfect priors, achieving state-of-the-art performance and surpassing it when using suboptimal guidance. The performance gain was larger in more degraded settings.
  • Continuous Control Experiments: In deep RL settings, adaptive methods were more robust and consistent. While non-adaptive E2R sometimes performed well with expert priors, it suffered performance degradation and was unable to learn effectively in some environments. AE2R, on the other hand, showed more stable performance and was able to speed up learning even with expert teachers, suggesting that robustness is key in complex deep RL tasks.
  • Prior Weight Behavior: The adaptive methods learned to assign smaller prior weights to degraded states, supporting the intuition that the algorithm learns to down-weight unreliable advice.

Limitations and Future Work

  • Value Function Estimation: The adaptive methods require value function estimation to learn the prior weights. While the authors note this is common in policy gradient methods, it adds a component to the learning process.
  • Generalizing Priors: The paper focuses on "soft action priors." Future work could explore leveraging more general forms of prior knowledge, such as affordances or pre-programmed "bots," to further enhance sample efficiency in RL.
  • Computational Cost: While the adaptive methods improve robustness, they introduce additional learned parameters (for prior weights), which might increase computational overhead.
  • Hyperparameter Tuning: The performance of adaptive methods can still be sensitive to hyperparameters, as seen with the $C_w$ parameter in the deep RL experiments.

The authors suggest that their work opens doors to leveraging more general priors and has exciting applications in improving sample efficiency and reducing computational requirements in RL benchmarks.

Summary updated

BibTeX
@inproceedings{centa2023soft,
  author = {Matheus Centa and Philippe Preux},
  title = {{Soft Action Priors: Towards Robust Policy Transfer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i6.25850},
}
Figure