Abstract: Continual reinforcement learning (CRL) empowers RL agents with the ability to learn a sequence of tasks, accumulating knowledge learned in the past and using the knowledge for problem-solving or future task learning. However, existing methods often focus on transferring fine-grained knowledge across similar tasks, which neglects the multi-granularity structure of human cognitive control, resulting in insufficient knowledge transfer across diverse tasks. To enhance coarse-grained knowledge transfer, we propose a novel framework called MT-Core (Multi-granularity knowledge Transfer for Continual reinforcement learning). MT-Core has a key characteristic of multi-granularity policy learning: 1) a coarse-grained policy formulation for utilizing the powerful reasoning ability of the large language model (LLM) to set goals, and 2) a fine-grained policy learning through RL which is oriented by the goals. We also construct a new policy library (knowledge base) to store policies that can be retrieved for multi-granularity knowledge transfer. Experimental results demonstrate the superiority of the proposed MT-Core in handling diverse CRL tasks versus popular baselines.
Introduction
Continual Reinforcement Learning (CRL) represents a significant frontier in artificial intelligence, aiming to develop agents capable of learning a sequence of tasks by accumulating and leveraging past knowledge. A persistent challenge in this field is the "plasticity vs. stability" trade-off: the need to adapt to new tasks without forgetting previously acquired skills. Most existing CRL methodologies focus on fine-grained knowledge transfer, such as sharing neural network weights or policy parameters. While effective for highly similar tasks, these methods often fail when tasks exhibit diverse state spaces, action spaces, or objectives.
This paper introduces MT-Core, a framework inspired by the hierarchical structure of human cognitive control. In humans, high-level reasoning (e.g., strategic planning in the prefrontal cortex) is decoupled from low-level motor control (e.g., precise execution in the cerebellum). MT-Core implements this multi-granularity structure by combining the reasoning capabilities of Large Language Models (LLMs) with the execution power of Reinforcement Learning (RL).
The MT-Core Framework
The MT-Core framework operates through a dual-layer architecture designed to facilitate both coarse-grained and fine-grained knowledge transfer. Unlike traditional methods that struggle with heterogeneous tasks, MT-Core uses language as a bridge for abstraction, allowing concepts learned in simple environments to be applied to complex ones.
Coarse-Grained Policy Formulation
At the high level, an LLM serves as the cognitive controller. This layer is responsible for "Coarse-Grained Policy Formulation." The process begins with a Reporter module that translates the current environment state $s_t$ into a textual description $e$. This description, along with retrieved experiences from a Policy Library, is fed into the LLM.
The LLM formulates a strategic plan $d$, which is then decomposed into a sequence of sub-goals $(g_1, \dots, g_n)$. Each goal $g_i$ includes a semantic description $s_g$, an intrinsic reward value $r_g$, and a verification function $v_g$ (e.g., check_black_key_is_carried). This high-level planning significantly reduces the exploration space for the RL agent by providing a logical roadmap.
Fine-Grained Policy Learning
The low-level layer handles "Fine-Grained Policy Learning." A goal-oriented RL agent receives the sub-goals from the LLM and interacts with the environment to learn the specific actions $a_t$ required to achieve them. The agent's learning is guided by an intrinsic reward function $R'(s, a, s', g)$ defined by the LLM's plan, in addition to any extrinsic rewards $r_t$ from the environment.
Policy Library and Reflexion Mechanism
A central component of MT-Core is the Policy Library, a knowledge base that stores both textual descriptions of successful strategies and the corresponding fine-grained policy weights. When faced with a new task, the system uses vector embeddings to retrieve the most relevant past experiences.
Furthermore, MT-Core incorporates a Reflexion Mechanism. If the RL agent fails to achieve a goal, the LLM analyzes performance feedback to pinpoint the cause of failure. It then iteratively adjusts the high-level plan, mimicking the human ability to learn from mistakes and refine strategies.
Experimental Evaluation
The framework was evaluated using the MiniGrid environment suite across two primary scenarios:
- Sequential Tasks: Four tasks of increasing difficulty (Task 1 to Task 4).
- Heterogeneous Tasks: A challenging setup where agents learn in a small state space (Task 1) and must transfer that knowledge to significantly larger, more complex grids (Tasks L1–L4).
MT-Core was compared against several CRL baselines, including PackNet, Fine-Tuning (FT), and FT-L2.
Results and Discussion
Experimental results highlight the significant advantages of multi-granularity transfer:
- Superior Performance: MT-Core achieved an average performance score ($A_N$) of 0.65, more than doubling the performance of the best baseline (Single-Task at 0.30).
- Positive Forward Transfer: MT-Core was the only method to demonstrate positive forward transfer ($FW = 0.17$), indicating that learning previous tasks directly facilitated the acquisition of new skills. All other baselines exhibited negative transfer, where previous knowledge interfered with new learning.
- Heterogeneous Success: MT-Core successfully transferred knowledge from small grids to large ones. By abstracting the concept of "finding a key" into language, the agent could apply the strategy regardless of the underlying mathematical state-space representation.
Conclusion
MT-Core demonstrates that multi-granularity knowledge transfer is essential for effective Continual Reinforcement Learning across diverse tasks. By using LLMs for high-level reasoning and RL for low-level execution, the framework overcomes the limitations of traditional fine-grained transfer. Future work will explore scaling this approach to continuous action spaces and managing larger, more complex policy libraries.
Summary updated