TL;DR
GAPD (Gold-Action Policy Distillation) introduces dense, token-level guidance into outcome-based reinforcement learning for agentic Knowledge Base Question Answering (KBQA). By dynamically aligning ground-truth "gold" actions to on-policy student states using intermediate execution entities, it constructs a self-teacher that corrects fine-grained errors during exploration. This framework achieves state-of-the-art performance across major benchmarks using a compact 8B parameter model, while actually reducing the number of required database interactions at inference time.
What Problem Does This Paper Address?
Agentic KBQA treats question answering as an interactive task. Instead of generating a complex query in one shot, an LLM agent emits a sequence of structured actions (e.g., Find_relation, Merge, Compare), receives feedback from a Knowledge Base (KB) executor, and iteratively navigates the graph to find the answer.
Currently, these models are optimized using Reinforcement Learning (like GRPO) driven by outcome-only rewards—meaning the model is judged solely on whether the final answer is correct. This creates a severe credit assignment problem. If an agent makes a single wrong relation choice early in a multi-turn trajectory, the entire rollout fails. The blanket negative reward tells the model that it failed, but not where or which token caused the mistake.
While ground-truth ("gold") logical forms are available in standard datasets, existing methods only use them for offline warm-starting or Supervised Fine-Tuning (SFT), completely discarding them during the on-policy RL phase. GAPD bridges this gap by turning gold actions into dense, real-time training guidance.
Methods
The GAPD framework enhances standard outcome-driven RL with three core components:
- Entity-Anchor Matching: Because an agent's on-policy exploration can easily deviate from the exact gold path, matching actions by turn index doesn't work. Instead, GAPD aligns states based on execution consequences. It treats the intermediate entities (Machine IDs or MIDs) reached during execution as anchors, computing the Jaccard similarity between the student's explored entity set and the gold states to find the most compatible alignment.
- Gold-Action-Conditioned Self-Teacher: Once a student state matches a gold state, the next valid gold action is appended to the student's context as a hidden hint. The current policy is then evaluated on this hinted prompt to act as a teacher distribution. Because the teacher is just the current model under valid procedural conditioning (rather than a separate frozen oracle), it preserves soft token preferences and avoids severe distribution shifts.
- Token-Level Fusion: The log-probability discrepancy between the self-teacher and the unhinted student policy forms a token-level advantage. This local guidance is fused directly into the trajectory-level GRPO advantage, applying a localized suppressive correction specifically on the divergent token spans.
Datasets
The authors evaluated GAPD on three widely recognized KBQA benchmarks using Freebase as the backend:
- GrailQA: Focuses heavily on compositional generalization, containing over 64,000 questions split into i.i.d., compositional, and zero-shot settings.
- WebQSP: A standard benchmark containing 4,737 questions with annotated semantic parses.
- GraphQ: Features 5,166 complex, graph-structured questions.
Key Insights and Ideas
- Self-Teaching vs. Hard Cloning: Traditional behavior cloning forces a model to mimic exact gold text strings, which can harm exploration. GAPD's self-teacher approach allows the model to learn from its own distribution under better conditioning, preserving alternative valid paths.
- State-Based Alignment: In interactive environments, execution state matters more than time. Matching trajectories via intermediate entity sets ensures that even if the model takes a roundabout path, it receives structurally sound guidance.
- Localized Suppressive Correction: Rather than shifting the entire policy distribution uniformly, token-level advantages act like a sharp brake precisely where the model begins to wander down an incorrect relation branch.
Key Results
GAPD (built on a Llama-3.1-8B backbone) consistently outperformed previous fine-tuning methods and even surpassed massive commercial LLMs (like Claude 4.6 Sonnet and GPT-5.3-Codex) plugged into the same interaction harness.
- GrailQA: Achieved an overall F1 score of 89.5%, a +3.9% relative improvement over the strong KBQA-R1 baseline. Notable gains were observed in the challenging zero-shot (+4.1%) and compositional (+5.5%) splits.
- WebQSP & GraphQ: Reached 86.1% and 55.1% F1 respectively, setting new state-of-the-art marks.
- Interaction Efficiency: GAPD achieved higher accuracy while requiring fewer environment calls on average (2.68 calls/query vs. 2.91 for KBQA-R1), proving that dense supervision yields more concise and precise graph navigation.
Limitations and Future Work
The primary limitation of GAPD is training overhead. Computing the self-teacher distributions requires scoring student tokens on supervised spans, making GAPD training roughly 1.5× slower than outcome-only RL. However, the authors emphasize that this is strictly a training-time trade-off; it introduces zero computational or memory overhead during inference.
For future work, extending this state-consequence matching framework to open-ended tool-use environments or web-navigation agents remains a promising direction.
Summary updated