GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering

Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Bowen Song, Weiqiang Wang, Zilei Wang, Liang WangarXiv 2026 · arXiv

TL;DR

GAPD (Gold-Action Policy Distillation) introduces dense, token-level guidance into outcome-based reinforcement learning for agentic Knowledge Base Question Answering (KBQA). By dynamically aligning ground-truth "gold" actions to on-policy student states using intermediate execution entities, it constructs a self-teacher that corrects fine-grained errors during exploration. This framework achieves state-of-the-art performance across major benchmarks using a compact 8B parameter model, while actually reducing the number of required database interactions at inference time.

What Problem Does This Paper Address?

Agentic KBQA treats question answering as an interactive task. Instead of generating a complex query in one shot, an LLM agent emits a sequence of structured actions (e.g., Find_relation, Merge, Compare), receives feedback from a Knowledge Base (KB) executor, and iteratively navigates the graph to find the answer.

Currently, these models are optimized using Reinforcement Learning (like GRPO) driven by outcome-only rewards—meaning the model is judged solely on whether the final answer is correct. This creates a severe credit assignment problem. If an agent makes a single wrong relation choice early in a multi-turn trajectory, the entire rollout fails. The blanket negative reward tells the model that it failed, but not where or which token caused the mistake.

While ground-truth ("gold") logical forms are available in standard datasets, existing methods only use them for offline warm-starting or Supervised Fine-Tuning (SFT), completely discarding them during the on-policy RL phase. GAPD bridges this gap by turning gold actions into dense, real-time training guidance.

Methods

The GAPD framework enhances standard outcome-driven RL with three core components:

  1. Entity-Anchor Matching: Because an agent's on-policy exploration can easily deviate from the exact gold path, matching actions by turn index doesn't work. Instead, GAPD aligns states based on execution consequences. It treats the intermediate entities (Machine IDs or MIDs) reached during execution as anchors, computing the Jaccard similarity between the student's explored entity set and the gold states to find the most compatible alignment.
  2. Gold-Action-Conditioned Self-Teacher: Once a student state matches a gold state, the next valid gold action is appended to the student's context as a hidden hint. The current policy is then evaluated on this hinted prompt to act as a teacher distribution. Because the teacher is just the current model under valid procedural conditioning (rather than a separate frozen oracle), it preserves soft token preferences and avoids severe distribution shifts.
  3. Token-Level Fusion: The log-probability discrepancy between the self-teacher and the unhinted student policy forms a token-level advantage. This local guidance is fused directly into the trajectory-level GRPO advantage, applying a localized suppressive correction specifically on the divergent token spans.
Figure 1: Overview of GAPD. GRPO supplies supervision mainly through the final answer reward. GAPD uses ENTITY-ANCHOR MATCHING to align an on-policy student state with a gold execution state, conditions the current policy on the aligned gold action to obtain a self-teacher distribution, assigns dense guidance to the selected student tokens, and fuses this guidance with the original GRPO advantage.

Datasets

The authors evaluated GAPD on three widely recognized KBQA benchmarks using Freebase as the backend:

  • GrailQA: Focuses heavily on compositional generalization, containing over 64,000 questions split into i.i.d., compositional, and zero-shot settings.
  • WebQSP: A standard benchmark containing 4,737 questions with annotated semantic parses.
  • GraphQ: Features 5,166 complex, graph-structured questions.

Key Insights and Ideas

  • Self-Teaching vs. Hard Cloning: Traditional behavior cloning forces a model to mimic exact gold text strings, which can harm exploration. GAPD's self-teacher approach allows the model to learn from its own distribution under better conditioning, preserving alternative valid paths.
  • State-Based Alignment: In interactive environments, execution state matters more than time. Matching trajectories via intermediate entity sets ensures that even if the model takes a roundabout path, it receives structurally sound guidance.
  • Localized Suppressive Correction: Rather than shifting the entire policy distribution uniformly, token-level advantages act like a sharp brake precisely where the model begins to wander down an incorrect relation branch.

Key Results

GAPD (built on a Llama-3.1-8B backbone) consistently outperformed previous fine-tuning methods and even surpassed massive commercial LLMs (like Claude 4.6 Sonnet and GPT-5.3-Codex) plugged into the same interaction harness.

  • GrailQA: Achieved an overall F1 score of 89.5%, a +3.9% relative improvement over the strong KBQA-R1 baseline. Notable gains were observed in the challenging zero-shot (+4.1%) and compositional (+5.5%) splits.
  • WebQSP & GraphQ: Reached 86.1% and 55.1% F1 respectively, setting new state-of-the-art marks.
  • Interaction Efficiency: GAPD achieved higher accuracy while requiring fewer environment calls on average (2.68 calls/query vs. 2.91 for KBQA-R1), proving that dense supervision yields more concise and precise graph navigation.
Figure 2: Training dynamics and interaction efficiency. Left: reward curves comparing w/o GAPD and w/ GAPD runs; each curve plots mean reward (outcome reward + format reward), with faint per-step values and EMA-smoothed values using α = 0.2. Right: interaction-efficiency comparison, reporting average F1 over three benchmarks against average interactions or calls per query on a log scale.

Limitations and Future Work

The primary limitation of GAPD is training overhead. Computing the self-teacher distributions requires scoring student tokens on supervised spans, making GAPD training roughly 1.5× slower than outcome-only RL. However, the authors emphasize that this is strictly a training-time trade-off; it introduces zero computational or memory overhead during inference.

For future work, extending this state-consequence matching framework to open-ended tool-use environments or web-navigation agents remains a promising direction.

Summary updated

Abstract

Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose token distribution is distilled back to the ordinary student policy over generated action-token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.
BibTeX
@misc{sun2026gapd,
  author = {Xin Sun and Jianan Xie and Zhongqi Chen and Qiang Liu and Shu Wu and Bowen Song and Weiqiang Wang and Zilei Wang and Liang Wang},
  title = {{GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering}},
  year = {2026},
  eprint = {2605.29584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.29584},
}
Figure