Introduction
In practical Question Answering (QA) scenarios, such as those encountered by virtual assistants like Alexa or Siri, providing a single correct answer is paramount. Most current Knowledge Graph (KG) based QA systems are evaluated using metrics like hits@k or Mean Reciprocal Rank (MRR), which assume an answer always exists within the graph. However, when an answer is missing or the reasoning path is ambiguous, these systems are often "forced" to guess. This research addresses this limitation by allowing agents to learn when not to answer, significantly boosting precision and reliability.
Methodology
The authors formulate the QA task as a Markov Decision Problem (MDP) where an agent traverses a Knowledge Graph to locate an answer entity. To move beyond binary right/wrong rewards, they introduce three primary innovations.
Ternary Reward Structure and the "No Answer" Node
The core of the approach is a three-tiered reward system:
- Positive ($r_{pos}$): Awarded for reaching the correct answer entity.
- Neutral (0): Awarded for choosing to abstain via a "no answer" action.
- Negative ($r_{neg}$): Awarded for reaching an incorrect entity.
To facilitate this, a synthetic "no answer" relation and entity are added to every node in the Knowledge Graph. This allows the agent to "exit" the search at any point if it determines that a high-confidence path to a correct answer is unavailable, effectively suppressing the "spurious paths" that lead to wild guesses.
Pre-training with Depth-First Search (DFS)
To overcome the "cold start" problem in massive search spaces, the model is pre-trained using imitation learning. Unlike previous methods that used Breadth-First Search (BFS)—which biases models toward shorter paths—this approach utilizes DFS to discover more diverse and complex multi-hop reasoning chains before the RL agent begins exploration.
Evaluation and Metrics
The model was evaluated on the FB15k-237 benchmark and a larger Alexa69k-378 dataset. Because traditional metrics do not account for abstention, the authors introduce the QA Score: the harmonic mean of Precision and Answer Rate. This metric acts like an F-score, balancing the system's desire to answer many questions with its need to be correct when it does.
Results
The introduction of the ternary reward led to a significant shift in agent behavior. On the Alexa69k-378 dataset, the "No Answer RL" model improved precision from 16.77% (standard RL) to 38.92%. While the overall Answer Rate decreased as the model learned a confidence threshold, the quality of the provided answers was substantially higher.
As shown in the results, the "All" model—combining supervised pre-training, RL, and the ternary reward—achieved the highest QA Score of 0.4213, outperforming existing state-of-the-art baselines by significant margins.
Conclusion
By allowing RL agents the option to remain silent, this research provides a framework for more trustworthy QA systems. The ternary reward structure effectively punishes logical leaps that don't represent valid factual relationships, making it a vital step toward deploying KG-based reasoning in production environments where precision is critical.
Summary updated