DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection

Nicolaas Jedema, Thuy Vu, Manish Gupta, Alessandro MoschittiarXiv 2022 · DOI · arXiv

Abstract

While transformers demonstrate impressive performance on many knowledge intensive (KI) tasks, their ability to serve as implicit knowledge bases (KBs) remains limited, as shown on several slot-filling, question-answering (QA), fact verification, and entity-linking tasks. In this paper, we implement an efficient, data-programming technique that enriches training data with KB-derived context and improves transformer utilization of encoded knowledge when fine-tuning for a particular QA task, namely answer sentence selection (AS2). Our method outperforms state of the art transformer approach on WikiQA and TrecQA, two widely studied AS2 benchmarks, increasing by 2.0% p@1, 1.3% MAP, 1.1% MRR, and 4.4% p@1, 0.9% MAP, 2.4% MRR, respectively. To demonstrate our improvements in an industry setting, we additionally evaluate our approach on a proprietary dataset of Alexa QA pairs, and show increase of 2.3% F1 and 2.0% MAP. We additionally find that these improvements remain even when KB context is omitted at inference time, allowing for the use of our models within existing transformer workflows without additional latency or deployment costs.

Introduction

Large Language Models (LLMs) like RoBERTa are frequently characterized as "implicit knowledge bases," yet they often struggle with reliability in knowledge-intensive (KI) tasks. A primary challenge is Answer Sentence Selection (AS2), where a model must identify the correct answer from a list of candidates. Standard models frequently fail to connect related entities—such as recognizing the relationship between "David Furnish" and "Elton John"—without explicit external guidance.

Existing solutions, such as Retrieval-Augmented Generation (RAG) or specialized KB-embeddings, introduce significant computational overhead and latency, making them difficult to deploy in real-time environments like voice assistants. The DP-KB approach addresses this by baking knowledge into the model during the training phase, allowing for high-performance inference without the need for external lookups at runtime.

Methodology: The DP-KB Pipeline

The researchers developed a novel data enrichment pipeline called DP-KB (Data Programming with Knowledge Bases). The process follows a structured workflow to enhance the model's internal representation of facts:

  1. Entity Tagging: An ElasticSearch index containing over 20 million KB items is used to identify and tag entities within both questions and answer candidates.
  2. Intersection Filtering: To provide a strong supervision signal, the system only adds KB context if an entity appears in both the question and the answer. This "intersection" acts as a bridge, helping the model focus on the most relevant factual links.
  3. Verbalization: KB metadata (e.g., categories like "celebrity") is converted into natural language and prepended or appended to the input string using standard separator tokens.
  4. Transfer-and-Adapt (TANDA): The model undergoes a two-step fine-tuning process. It is first trained on a large general dataset (ASNQ) enriched with KB context, then adapted to specific target benchmarks like WikiQA or TrecQA.
The DP-KB pipeline illustrating the transfer from general QA (ASNQ) to specific task adaptation (AlexaQA, WikiQA, TrecQA) using KB-enriched inputs.

Experimental Results

The DP-KB approach was evaluated against the state-of-the-art TANDA method across several benchmarks, including WikiQA, TrecQA, and a proprietary AlexaQA dataset.

Performance Gains

On the TrecQA benchmark, the model saw a substantial increase of 4.4% in Precision@1 and 2.4% in Mean Reciprocal Rank (MRR). Similar improvements were noted in WikiQA, with a 2.0% boost in P@1. In the industry-scale AlexaQA dataset, which includes 107,000 unique real-world questions, the method achieved a 2.3% increase in F1 score.

Comparative results on the TrecQA benchmark showing that DP-KB (Prepend, Intersection) consistently outperforms the baseline, even when KB context is omitted at inference.

Inference Efficiency

A critical finding of this research is that the performance gains persist even when the KB context is omitted at inference time. This suggests that the KB enrichment during training acts as a "knowledge catalyst," teaching the model how to better attend to factual relationships within its own weights. This allows the model to be deployed as a plug-and-play replacement in existing workflows with zero added latency.

Conclusion and Future Work

DP-KB demonstrates that data programming with knowledge bases can significantly improve transformer performance on KI tasks without increasing production costs. While highly effective, the method's success is currently tied to the coverage of the underlying KB. Future research will focus on further disentangling language acquisition from knowledge acquisition to create models that can learn facts more robustly across diverse linguistic variations.

Summary updated

Abstract

While transformers demonstrate impressive performance on many knowledge intensive (KI) tasks, their ability to serve as implicit knowledge bases (KBs) remains limited, as shown on several slot-filling, question-answering (QA), fact verification, and entity-linking tasks. In this paper, we implement an efficient, data-programming technique that enriches training data with KB-derived context and improves transformer utilization of encoded knowledge when fine-tuning for a particular QA task, namely answer sentence selection (AS2). Our method outperforms state of the art transformer approach on WikiQA and TrecQA, two widely studied AS2 benchmarks, increasing by 2.0% p@1, 1.3% MAP, 1.1% MRR, and 4.4% p@1, 0.9% MAP, 2.4% MRR, respectively. To demonstrate our improvements in an industry setting, we additionally evaluate our approach on a proprietary dataset of Alexa QA pairs, and show increase of 2.3% F1 and 2.0% MAP. We additionally find that these improvements remain even when KB context is omitted at inference time, allowing for the use of our models within existing transformer workflows without additional latency or deployment costs.
BibTeX
@misc{jedema2022dp,
  author = {Nicolaas Jedema and Thuy Vu and Manish Gupta and Alessandro Moschitti},
  title = {{DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection}},
  year = {2022},
  eprint = {2203.09598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.09598},
}
Figure