Toward Robust Long Range Policy Transfer

Wei-Cheng Tseng, Jin-Siang Lin, Yao-Min Feng, Min SunAAAI 2021 · DOI

Introduction

Reinforcement learning (RL) agents typically struggle with sample efficiency, often requiring millions of interactions to learn tasks from scratch. While transfer learning—reusing knowledge from a pre-training task for a new target task—offers a potential solution, it is frequently hampered by two significant issues: biased exploration and dynamics shifts. Biased exploration occurs when a pre-trained policy's prior knowledge prevents it from exploring new, high-reward regions (e.g., an agent trained to run forward struggles to learn to run backward). Dynamics shifts involve changes in the environment, such as gravity or terrain, that render old skills ineffective.

This paper introduces a hierarchical RL framework designed specifically for "long-range" policy transfer, where the target task differs substantially from the pre-training environment.

Proposed Method: Hierarchical Multiplicative Policies

The authors propose a hierarchical architecture that decomposes behavior into a set of specialized "primitive" policies.

Architecture

As shown in Figure 1, the model consists of a combination function ($C_\phi$) and a set of $n$ primitive policies ($\pi_{\theta_1} \dots \pi_{\theta_n}$). For a given state $s_t$ and goal $g_t$, the combination function outputs a weight vector $w_{1:n}$. These weights are then used to combine the outputs of the primitives via a multiplicative rule to produce the final policy $\Pi(a_t|s_t, g_t)$.

Hierarchical architecture and long-range task visualization

Pre-training with Regularization

To ensure the primitives are diverse and collectively useful, the authors introduce two regularization terms during pre-training:

  • Diversity Regularization (DR): This term uses Jensen-Shannon divergence to force primitives to learn distinct behaviors, preventing them from collapsing into a single "average" skill.
  • Utility Regularization (UR): By maximizing the entropy of the primitive weights, UR ensures that all primitives are utilized during training, avoiding over-reliance on a subset of skills.

Alternating Transfer Strategy

A key innovation of this work is the alternating update schedule used during the transfer phase. Instead of fine-tuning the entire network simultaneously, which often leads to catastrophic forgetting, the process alternates between two steps:

  1. Step A (Freeze Primitives): Only the combination function is trained. This allows the agent to learn how to re-sequence and re-weight existing skills for the new task without altering the skills themselves.
  2. Step B (Freeze Combination): The combination function is fixed while the primitives are fine-tuned. This enables the underlying skills to adapt to new environmental dynamics (e.g., different friction or slopes).

This cycle balances the stable reuse of prior knowledge with the flexibility needed for adaptation.

Experimental Evaluation

The framework was evaluated using several continuous control tasks in the PyBullet physics engine, focusing on scenarios with significant task shifts.

Evaluation environments

The datasets included:

  • Ant: Moving from a "forward" goal distribution to a "backward" one (a 180-degree shift).
  • 2D Walker: Transitioning from flat terrain to complex slopes and cliffs.
  • HalfCheetah: Moving from jumping over simple walls to navigating "terraced" environments requiring sequential climbing.

Key Results and Insights

The proposed method demonstrated superior performance and sample efficiency compared to established baselines like Multiplicative Compositional Policies (MCP) and Meta-Learning Shared Hierarchies (MLSH).

  • Robustness to Task Shift: In the Ant "Long Range" task, the hierarchical approach successfully adapted to the backward goal, whereas standard fine-tuning methods often failed due to exploration bias.
  • Efficiency: The agent reached high performance levels significantly faster than training from scratch, saving millions of environment steps.
  • Adaptability: The alternating strategy proved robust even when pre-trained primitives were sub-optimal, showing that the framework can effectively "repair" and adapt skills to new dynamics.

Conclusion

By combining diverse primitives with an alternating training schedule, this framework provides a robust path for long-range policy transfer in RL. Future work may focus on automating the selection of the number of primitives and exploring task-agnostic skill learning in more complex domains.

Summary updated

BibTeX
@inproceedings{tseng2021toward,
  author = {Wei-Cheng Tseng and Jin-Siang Lin and Yao-Min Feng and Min Sun},
  title = {{Toward Robust Long Range Policy Transfer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i11.17196},
}
Figure