TL;DR
Knowledge-Grounded Reinforcement Learning (KGRL) introduces an attention-based actor architecture that lets agents dynamically consult, combine, and reuse external guidelines alongside their own internal policy. By decoupling knowledge representation (learnable key embeddings) from selection (a state-dependent query), agents achieve higher sample efficiency, better zero-shot generalization, and interpretable decision-making across discrete and continuous control tasks.
What Problem Does This Paper Address?
Humans rarely learn complex skills entirely from scratch; we consult manuals, follow rules, and adapt advice from multiple sources while developing our own intuition. In contrast, standard reinforcement learning (RL) typically starts tabula rasa.
Existing methods that incorporate external signals face severe bottlenecks:
- Imitation Learning & Learning from Demonstrations (LfD): Highly data-hungry, prone to compounding errors, and rigidly tied to the specific task distribution.
- Hierarchical & Program-Guided RL: Rely on rigid subgoal hierarchies or explicit control programs, making it difficult to rearrange or transfer learned sub-skills to new environments.
- Transfer & Meta-RL: Require massive pre-training data across task distributions or complex distillation pipelines.
The authors formulate KGRL to answer three core questions: How can an agent learn to consult imperfect external knowledge from disparate sources? How can it efficiently develop its own internal policy in tandem? And how can knowledge representations be decoupled so they can be recombined, rearranged, or reused without retraining?
Methods
The paper introduces an actor architecture compatible with standard policy-gradient and actor-critic algorithms (e.g., PPO, SAC).
+-----------------------------+
| State Observation |
+--------------+--------------+
|
+----------------+----------------+
| |
+--------v-------+ +-------v--------+
| Inner Actor | | Query Network |
| pi_in , k_in(s)| | u(s) |
+--------+-------+ +-------+--------+
| |
| Dot-Product |
| <==============================>|
| Attentions |
| <==============================>|
| |
+--------+-------+ +-------+--------+
| Knowledge Keys | <---------------+ External Rules |
| k_g1, k_g2 | | pi_g1, pi_g2 |
+----------------+ +----------------+
- Decoupled Key-Query Mechanism:
- External Knowledge Keys ($k_{g_j} \in \mathbb{R}^{d_k}$): Each external rule or heuristic mapping $\pi_{g_j}: \mathcal{S} \to \mathcal{A}$ is assigned an order-invariant, learnable embedding key that represents the mapping itself rather than a single state-action pair.
- Inner Actor ($\pi_{in}, k_{in}$): The agent learns its own internal policy $\pi_{in}(s)$ via network $\phi_{in}(s)$ and a state-dependent inner key $k_{in}(s) = \phi_k(s) \in \mathbb{R}^{d_k}$.
- Query ($\mathbf{u}$): A learned function $\phi_u(s)$ emits a state-dependent query $u \in \mathbb{R}^{d_k}$. Attention weights are computed via dot product: $w_{in} = u \cdot k_{in}$ and $w_{g_j} = u \cdot k_{g_j}$.
- Action Selection:
- Discrete Action Spaces: A standard softmax-weighted sum of action probability distributions across all knowledge sources and the inner policy.
- Continuous Action Spaces: Weighted sums can cause catastrophic interpolation (e.g., averaging two avoidance actions might steer directly into an obstacle). Instead, the model selects an active knowledge module using Gumbel-Softmax ($e \sim \text{Gumbel-Softmax}(\mathbf{w})$) and samples an action via the reparameterization trick, keeping the actor differentiable end-to-end.
Datasets
Evaluations cover both discrete gridworlds and continuous robotics manipulation environments:
- MiniGrid (Discrete): Grid navigation with sparse rewards:
Empty-5x5,Empty-Random-5x5,Empty-16x16,Unlock,DoorKey-5x5, andDoorKey-8x8. Knowledge sources include simple directional heuristics: KG1 (pick up key), KG2 (open door), and KG3 (navigate to goal). - OpenAI Robotics (Continuous): Multi-goal environments with sparse binary rewards:
Fetch ReachandFetch Pick-And-Place. Knowledge sources specify rough directional vectors toward the target object or goal.
Key Insights and Ideas
- Knowledge Modular Disentanglement: Because keys are task- and position-independent vectors in a shared embedding space, policies can be added, dropped, or swapped zero-shot at inference time without breaking the query network.
- Knowledge Distillation via Co-learning: The inner actor doesn't just defer to external knowledge; it mimics and internalizes it. Even when all external heuristics are stripped away at test time, the inner actor alone maintains peak performance.
- Robustness to Redundant/Imperfect Rules: Irrelevant or noisy knowledge sources do not destabilize learning—the query network simply learns near-zero attention weights for unhelpful keys.
Key Results
- Sample Efficiency in Sparse Regimes: In MiniGrid
DoorKey-8x8, baseline PPO fails completely (0.0 average reward across 3M steps), whereas PPO-KGRL reaches full convergence. In robotics, SAC-KGRL converges faster and with lower variance than SAC or SAC with Behavior Cloning (SAC-BC). - Zero-Shot Policy Transfer: Agents trained on
DoorKey-8x8transfer zero-shot toUnlock: PPO scores 0.00, while PPO-KGRL achieves an average return of 0.85. - Internalization: In
Pick-And-Place, stripping all external knowledge at test time leaves the standalone inner actor operating at a 0.99 success rate (identical to the full ensemble). - Interpretability: Plotting unnormalized attention weights over time recovers distinct, intuitive behavioral phases (e.g., sequentially attending to key retrieval $\to$ door opening $\to$ goal search $\to$ goal reaching).
Limitations and Future Work
- Hand-Crafted Knowledge Priors: Current experiments rely on manually designed rules or pre-trained policies with pre-aligned state-action representations.
- Action/State Space Alignment: The framework assumes external policies share the target environment's state and action spaces.
- Strict Constraint Enforcement: Attention weights softly arbitrate knowledge selection, which is unsuitable for hard safety boundaries or invariant rules.
- Future Directions: Automating rule extraction directly from unstructured sources (e.g., web text, manuals), resolving logical conflicts among competing knowledge modules, and handling conditionally dependent rules.
Summary updated