Hi-Core: Hierarchical Knowledge Transfer for Continual Reinforcement Learning

Chaofan Pan, Xin Yang, Hao Wang, Wei Wei, Tianrui Li

TL;DR

This paper introduces MT-Core, a novel framework for Continual Reinforcement Learning (CRL) that leverages Large Language Models (LLMs) to facilitate multi-granularity knowledge transfer. By using LLMs for high-level goal formulation and RL for low-level action execution, MT-Core significantly improves an agent's ability to learn across diverse tasks, outperforming existing methods.

What Problem Does This Paper Address?

Real-world applications for Reinforcement Learning (RL) agents demand the ability to learn continuously from a sequence of tasks without forgetting past knowledge. This is the domain of Continual Reinforcement Learning (CRL). While current CRL methods can transfer knowledge, they often struggle when tasks are significantly different, typically focusing on fine-grained knowledge transfer (like sharing policy networks). This approach is insufficient for diverse tasks because it neglects the hierarchical, multi-granularity structure of human cognitive control, which involves both high-level planning and low-level execution. The paper aims to bridge this gap by enabling more effective coarse-grained knowledge transfer across diverse tasks.

Methods

MT-Core introduces a two-layer approach:

  1. Coarse-Grained Policy Formulation: This layer utilizes the reasoning capabilities of an LLM. The LLM takes prompts, which can include retrieved experiences from a policy library, and formulates a sequence of high-level goals. Each goal is described textually and includes an intrinsic reward function to guide the agent. This process is iterative, with the LLM refining its goals based on feedback from the agent's performance.
  2. Fine-Grained Policy Learning: This layer uses goal-oriented RL. A standard RL model learns a low-level policy that aims to achieve the goals set by the LLM. The agent's intrinsic reward is defined based on whether it achieves these specific goals, providing a more granular signal than just the environment's external reward.

A Policy Library is central to MT-Core. It stores successful coarse-grained policies (textual goal sequences) and their corresponding fine-grained policies (learned RL models). When a new task arrives, MT-Core retrieves relevant experiences from this library to inform the LLM's goal formulation, thereby enhancing knowledge transfer. This approach is inspired by the "Reflexion" concept, which uses LLMs for iterative policy refinement and self-correction.

Datasets

The experiments were conducted using a suite of MiniGrid environments. These environments are characterized by image-based state observations and discrete action spaces. The authors specifically crafted a sequence of four distinct tasks within the MiniGrid framework, increasing in difficulty to simulate a human-like learning progression and facilitate the tracking of knowledge transfer.

Key Insights and Ideas

  • Human Cognition as Inspiration: The paper argues that human cognitive control is inherently multi-granularity (hierarchical), involving both high-level planning and low-level execution. Mimicking this structure is key to effective knowledge transfer in CRL, especially across diverse tasks.
  • LLMs for High-Level Reasoning: LLMs are powerful tools for abstract reasoning and goal formulation. MT-Core leverages this to create high-level policies that guide the RL agent, moving beyond simple fine-tuning of low-level policies.
  • Policy Library for Experience Reuse: A dedicated policy library is crucial for storing and retrieving past successful strategies. This allows the agent to quickly adapt to new tasks by leveraging prior knowledge, rather than starting from scratch or relying on potentially unstable fine-tuning.
  • Coarse-Grained Transfer for Diversity: The core innovation is enabling coarse-grained knowledge transfer. This means transferring abstract goals and strategies, which is more robust and effective for diverse tasks than transferring low-level, task-specific policies.
  • Heterogeneous Task Applicability: The framework shows promise in handling heterogeneous tasks, where state spaces might differ, without requiring complex masking techniques.

Key Results

  • MT-Core significantly outperforms popular baselines (Single-Task, Fine-Tuning, Fine-Tuning with L2 Regularization, and PackNet) on diverse CRL tasks in the MiniGrid environment.
  • The average performance metric for MT-Core is more than double that of the best baseline.
  • MT-Core demonstrates positive forward transfer, meaning it effectively leverages knowledge from previous tasks to improve performance on new ones, unlike most baselines which show zero or negative forward transfer.
  • While MT-Core still experiences some performance degradation when transitioning between highly dissimilar tasks, it shows a rapid recovery, highlighting its adaptability.
  • Ablation studies confirm the importance of both the policy library (MT-Core-SG vs. MT-Core) and the feedback mechanism (MT-Core-Once vs. MT-Core) for achieving optimal performance.

Limitations and Future Work

  • Forgetting Mitigation: While MT-Core excels at forward transfer, the paper notes that it doesn't explicitly focus on minimizing forgetting. The authors suggest that fine-tuning strategies could be employed in situations where forgetting mitigation is a higher priority.
  • Task Similarity Assumption: The current framework is most effective when there are some abstract similarities that the LLM can leverage. The performance on extremely dissimilar tasks might still be a challenge.
  • Computational Cost: Integrating LLMs can be computationally intensive. While not explicitly detailed as a limitation, this is an inherent consideration for such approaches.

Future work could explore more sophisticated methods for balancing plasticity and stability, further enhancing the LLM's ability to handle even more diverse and complex task sequences, and investigating the framework's scalability to larger and more complex environments.

Summary updated

Abstract

The ability to learn continuously in dynamic environments is a crucial requirement for reinforcement learning (RL) agents applying in the real world. Despite the progress in continual reinforcement learning (CRL), existing methods often suffer from insufficient knowledge transfer, particularly when the tasks are diverse. To address this challenge, we propose a new framework, Hierarchical Continual reinforcement learning via large language model (Hi-Core), designed to facilitate the transfer of high-level knowledge. Hi-Core orchestrates a twolayer structure: high-level policy formulation by a large language model (LLM), which represents agenerates a sequence of goals, and low-level policy learning that closely aligns with goal-oriented RL practices, producing the agent's actions in response to the goals set forth. The framework employs feedback to iteratively adjust and verify highlevel policies, storing them along with low-level policies within a skill library. When encountering a new task, Hi-Core retrieves relevant experience from this library to help to learning. Through experiments on Minigrid, Hi-Core has demonstrated its effectiveness in handling diverse CRL tasks, which outperforms popular baselines.
BibTeX
@inproceedings{pan2025hi,
  author = {Chaofan Pan and Xin Yang and Hao Wang and Wei Wei and Tianrui Li},
  title = {{Hi-Core: Hierarchical Knowledge Transfer for Continual Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=AupKvx43p2},
}
Figure