Transfer Reinforcement Learning across Homotopy Classes

Zhangjie Cao, Minae Kwon, Dorsa Sadigh

Abstract

Standard fine-tuning in reinforcement learning (RL) often fails when a task requires a fundamental change in strategy—such as switching from swerving right to swerving left around an obstacle. This paper identifies that such failures occur because different strategies belong to distinct homotopy classes separated by high-loss reward barriers. We introduce Ease-In-Ease-Out (EIEO) fine-tuning, a method that bypasses these "unpassable" barriers by temporarily relaxing constraints and then gradually reintroducing them through curriculum learning.

The Problem: Homotopy Classes and the Continuity Trap

In supervised learning, fine-tuning a pre-trained model on a new dataset is a standard and effective practice. In Reinforcement Learning, however, fine-tuning is notoriously brittle. The authors identify a geometric reason for this failure: topological barriers in the reward landscape.

In topology, two paths are homotopic if one can be continuously deformed into the other without hitting a barrier. In RL, "barriers" are states with large negative rewards, such as walls or unsafe joint configurations. If a robot learns to reach a goal by going around a barrier to the right (Source Task), but the optimal path in a new environment is to the left (Target Task), standard fine-tuning fails.

Illustration of trajectories in different homotopy classes (H1, H2) separated by a barrier (Sb).

Because fine-tuning changes policy parameters continuously, the robot's trajectory must "pass through" the barrier to transition between these classes. During this transition, the robot hits the barrier, receives a massive negative reward, and the resulting gradients push the policy back to its original state. The agent is effectively "trapped" in its original homotopy class.

Methods: Ease-In-Ease-Out (EIEO)

The authors propose Ease-In-Ease-Out (EIEO) fine-tuning to bridge these disconnected strategy spaces. The algorithm consists of two main stages:

  1. Ease-In (Relaxing Stage): The agent "eases in" to the target goal by temporarily removing the negative reward associated with barriers. By ignoring the "walls," the policy can freely migrate from the source strategy to a trajectory that reaches the target goal, even if it currently overlaps with the barrier.
  2. Ease-Out (Curriculum Learning Stage): Once the agent reaches the target goal (even by "cheating" through barriers), the penalty is gradually reintroduced. This is achieved by either increasing the Reward Weight of the penalty from 0 to its full value or by starting with a tiny Barrier Set Size and slowly inflating it to original dimensions.

This approach ensures that at every step of the curriculum, the optimal policy remains "close" to the previous one, allowing the agent to find a path around the barrier without falling into a high-loss trap.

Key Insights and Results

The paper demonstrates that "barriers" are not just physical walls; they can represent joint limits in a Mujoco Ant or unsafe spoon angles in assistive feeding tasks. The continuity of policy parameters does not guarantee a smooth transition between trajectories in different homotopy classes, creating a ridge in the loss landscape that standard gradient descent cannot cross.

Loss landscapes in parameter space showing the high-loss barrier (ridge) that prevents standard fine-tuning from switching strategies.

EIEO was evaluated across several simulated robotics environments, including 2D Navigation, Lunar Lander, Fetch Reach (3D), Mujoco Ant, and Assistive Feeding. Key results include:

  • Sample Efficiency: In the Lunar Lander task, EIEO reached the target reward in ~80k steps, while naive fine-tuning failed to converge after 300k steps.
  • Success in Complex Tasks: In high-dimensional tasks like Assistive Feeding, EIEO was often the only method to successfully transfer knowledge to a new homotopy class, outperforming baselines like Progressive Neural Networks (PNN) and $L^2$-SP.
  • Robustness: The performance gap between EIEO and other methods widened as barrier sizes increased, highlighting the severity of the homotopy problem in constrained environments.

Limitations and Future Work

The primary limitation of EIEO is the requirement for a-priori knowledge of barrier locations to perform the relaxation. Future research may focus on automatically discovering homotopy classes from state-action data to enable the "Ease-In" stage without manual reward engineering. Additionally, extending the method to dynamic environments with moving barriers remains an open challenge.

Summary updated

Abstract

The ability for robots to transfer their learned knowledge to new tasks -- where data is scarce -- is a fundamental challenge for successful robot learning. While fine-tuning has been well-studied as a simple but effective transfer approach in the context of supervised learning, it is not as well-explored in the context of reinforcement learning. In this work, we study the problem of fine-tuning in transfer reinforcement learning when tasks are parameterized by their reward functions, which are known beforehand. We conjecture that fine-tuning drastically underperforms when source and target trajectories are part of different homotopy classes. We demonstrate that fine-tuning policy parameters across homotopy classes compared to fine-tuning within a homotopy class requires more interaction with the environment, and in certain cases is impossible. We propose a novel fine-tuning algorithm, Ease-In-Ease-Out fine-tuning, that consists of a relaxing stage and a curriculum learning stage to enable transfer learning across homotopy classes. Finally, we evaluate our approach on several robotics-inspired simulated environments and empirically verify that the Ease-In-Ease-Out fine-tuning method can successfully fine-tune in a sample-efficient way compared to existing baselines.
BibTeX
@article{cao2021transfer,
  author = {Zhangjie Cao and Minae Kwon and Dorsa Sadigh},
  title = {{Transfer Reinforcement Learning across Homotopy Classes}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2021},
  url = {https://openreview.net/forum?id=hmoSApIhyNR},
}
Figure