Hi-Core: Hierarchical Knowledge Transfer for Continual Reinforcement Learning

Chaofan Pan, Xin Yang, Hao Wang, Wei Wei, Tianrui Li

Abstract

Continual Reinforcement Learning (CRL) aims to develop agents capable of learning a sequence of tasks while accumulating knowledge. However, traditional methods often struggle with knowledge transfer across diverse or heterogeneous tasks where state spaces and goals differ. We introduce Hi-Core (also referred to as MT-Core), a novel framework that implements human-like hierarchical reasoning. By decoupling high-level "coarse-grained" strategic planning—handled by a Large Language Model (LLM)—from low-level "fine-grained" action execution, Hi-Core facilitates robust knowledge transfer even when mathematical input dimensions vary between tasks.

Introduction

The primary bottleneck in current CRL research is the reliance on fine-grained transfer, typically involving the sharing or adaptation of neural network weights. While effective for nearly identical tasks, this approach fails when tasks have different goals or observation spaces. Humans overcome this by transferring high-level strategies (e.g., "find a key to open a door") rather than specific motor sequences. Hi-Core replicates this multi-granularity approach to bridge the gap between mathematically incompatible tasks.

Methodology

Hi-Core splits the learning process into two distinct layers to manage complexity and facilitate transfer:

  1. Coarse-grained Policy Formulation (The "Brain"): An LLM serves as a high-level planner. It processes textual descriptions of the environment and retrieves relevant past experiences from a dedicated Policy Library to formulate a sequence of descriptive goals.
  2. Fine-grained Policy Learning (The "Muscles"): A goal-conditioned RL agent receives these high-level objectives and learns the specific action sequences (e.g., directional movements) required to achieve them.
Overview of the Hi-Core framework illustrating hierarchical transfer between coarse-grained planning and fine-grained execution.

Key Components

  • Policy Library: A repository storing successful past policies, including text-based plans and neural network weights. It utilizes vector embeddings to identify similar past environments for new task planning.
  • Iterative Refinement (Reflexion): The system incorporates a feedback loop where the RL agent's performance is reported back to the LLM. If a goal is not met, the LLM analyzes the failure and adjusts the high-level plan.
  • Language as a Bridge: Natural language serves as a universal interface, allowing the transfer of strategic "wisdom" between tasks that are incompatible at the weight-sharing level.
The CRL Agent architecture showing the interaction between the LLM planner, the policy library, and the RL environment.

Experimental Evaluation

The framework was evaluated using the MiniGrid benchmark, involving navigation, key retrieval, and goal reaching. The authors tested the agent on a sequence of four tasks of increasing difficulty and specifically designed heterogeneous versions where the agent's field of vision (state space) changed size between tasks.

Key Results

  • Performance Gains: Hi-Core achieved an average normalized return of 0.65, more than doubling the performance of the Single-Task baseline (0.30) and significantly outperforming established CRL methods like PackNet (0.20).
  • Positive Forward Transfer: Hi-Core was the only method to demonstrate strong positive Forward Transfer (0.17), indicating that knowledge from earlier tasks accelerated learning in subsequent ones.
  • Robustness to Heterogeneity: In experiments where input dimensions changed, Hi-Core successfully transferred strategies from small to large environments, whereas agents learning from scratch failed entirely.

Discussion and Limitations

While Hi-Core represents a significant step forward, it remains dependent on the reasoning capabilities of the underlying LLM. Furthermore, the system currently relies on a "Reporter" to translate environment states into text, which may be difficult to scale to complex 3D or real-world robotic environments. Future work will explore vision-language models to automate environment perception and reduce computational latency associated with LLM inference.

Conclusion

Hi-Core demonstrates that hierarchical, language-based reasoning is a powerful tool for Continual Reinforcement Learning. By separating strategy from execution, the framework avoids the "weight-sharing trap" and enables agents to accumulate and apply knowledge across diverse task sequences.

Summary updated

BibTeX
@inproceedings{pan2025hi,
  author = {Chaofan Pan and Xin Yang and Hao Wang and Wei Wei and Tianrui Li},
  title = {{Hi-Core: Hierarchical Knowledge Transfer for Continual Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=AupKvx43p2},
}
Figure