Abstract
Receiving knowledge, abiding by laws, and being aware of regulations are common behaviors in human society. Bearing in mind that reinforcement learning (RL) algorithms benefit from mimicking humanity, in this work, we propose that an RL agent can act on external guidance in both its learning process and model deployment, making the agent more socially acceptable. We introduce the concept, Knowledge-Grounded RL (KGRL), with a formal definition that an agent learns to follow external guidelines and develop its own policy. Moving towards the goal of KGRL, we propose a novel actor model with an embedding-based attention mechanism that can attend to either a learnable internal policy or external knowledge. The proposed method is orthogonal to training algorithms, and the external knowledge can be flexibly recomposed, rearranged, and reused in both training and inference stages. Through experiments on tasks with discrete and continuous action space, our KGRL agent is shown to be more sample efficient and generalizable, and it has flexibly rearrangeable knowledge embeddings and interpretable behaviors.
Introduction
Reinforcement Learning (RL) is notoriously "sample hungry," often requiring millions of trials to learn simple tasks. In contrast, humans are expert social learners; we do not just learn by trial and error, but by observing others and adapting external strategies to our own needs. This paper formalizes this behavior as Knowledge-Grounded Reinforcement Learning (KGRL), where an agent learns to integrate external "tips" or guidelines while developing its own internal policy.
The KIAN Architecture
To achieve the goals of KGRL, we introduce the Knowledge-Inclusive Attention Network (KIAN). Unlike previous methods that use fixed fusion mechanisms, KIAN is designed for flexibility and incremental learning.
KIAN's architecture is built on three pillars:
- The Inner Actor: A standard learnable RL policy that develops the agent's own unique strategy.
- Knowledge Keys (Embeddings): Each external policy is assigned a learnable embedding. Because these are independent of the specific state or action, the agent can rearrange, add, or swap policies like building blocks without retraining the entire network.
- Attentive Querying: The agent uses a state-dependent "query" to decide which policy to trust at any given moment through an embedding-based attentive action prediction mechanism.
Addressing the Entropy Imbalance
A significant contribution of this work is identifying and solving the Entropy Imbalance problem in Maximum Entropy RL. In these frameworks, agents are incentivized to be "uncertain" to encourage exploration. However, if an agent's internal policy is very uniform (high entropy), it can mathematically "drown out" useful external advice that is more specific (lower entropy). KIAN addresses this by modifying how policy distributions are fused, using a scaling variable ($\beta$) for discrete tasks and a lower-bound approximation for continuous tasks to prevent the agent from ignoring smart external advice.
Experimental Results and Generalization
We evaluated KIAN across two distinct domains: MiniGrid (discrete navigation) and OpenAI Robotics (continuous manipulation tasks like FetchPush and FetchPickAndPlace).
KIAN consistently demonstrated superior sample efficiency, reaching higher rewards faster than baselines such as A2T and KoGuN. Notably, KIAN showed remarkable Zero-Shot Generalization. For example, an agent trained on a $5\times5$ grid could be transferred to a $16\times16$ grid and maintain high performance by relying on its external navigation knowledge. Furthermore, in incremental learning scenarios, KIAN required significantly fewer samples to master new tasks by reusing previously acquired knowledge modules.
Conclusion
KIAN satisfies five essential properties for KGRL: it is knowledge-acquirable, sample-efficient, generalizable, compositional, and incremental. By separating policy representation from fusion logic, it allows for a more human-like, flexible approach to learning. Future work will explore filtering large libraries of potentially irrelevant knowledge and fusing policies across heterogeneous action spaces.
Summary updated