Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation

Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang

Abstract

Although pre-training models have achieved great success in dialogue generation, their performance drops dramatically when the input contains an entity that does not appear in pre-training and fine-tuning datasets (unseen entity). To address this issue, existing methods leverage an external knowledge base to generate appropriate responses. In real-world scenario, the entity may not be included by the knowledge base or suffer from the precision of knowledge retrieval. To deal with this problem, instead of introducing knowledge base as the input, we force the model to learn a better semantic representation by predicting the information in the knowledge base, only based on the input context. Specifically, with the help of a knowledge base, we introduce two auxiliary training objectives: 1) Interpret Masked Word, which conjectures the meaning of the masked entity given the context; 2) Hypernym Generation, which predicts the hypernym of the entity based on the context. Experiment results on two dialogue corpus verify the effectiveness of our methods under both knowledge available and unavailable settings.

Introduction

Large-scale pre-trained chatbots, such as Facebook’s Blender, are often "frozen in time." When users mention recent news events or niche technical terms not found in the original training corpus—referred to as "unseen entities"—these models frequently hallucinate or provide nonsensical answers.

The standard solution is Knowledge-Grounded Generation, where the model retrieves relevant snippets from a Knowledge Base (KB) like Wikipedia to use as extra input. However, this approach relies heavily on KB coverage and retrieval accuracy. If an entity is too new to be in the KB or if the retrieval system fetches the wrong information due to ambiguity, the model fails. This paper introduces KE-Blender, a fine-tuning approach that teaches models to infer the meaning of unknown words from context, making them robust even when external knowledge is unavailable at runtime.

Methodology: Knowledge Enhanced Fine-Tuning

Instead of treating external knowledge as a crutch to be used during inference, the authors propose using it as a "teacher" during the fine-tuning phase. By adding auxiliary tasks, the model is forced to internalize semantic patterns.

The KE-Blender framework: Training with auxiliary tasks vs. robust inference on unseen entities.

Using the Blender-90M model as a backbone, the researchers introduced two new objectives:

  1. Interpret Masked Word: The model receives a sentence with an entity masked (e.g., "I want to submit a paper to [MASK]") and must generate the definition of that entity (e.g., "EMNLP is a leading conference...") based solely on context.
  2. Hypernym Generation: The model learns to predict a broader category (hypernym) for an entity. For instance, given "EMNLP," the model learns to output "conference" using WordNet as ground truth.

These tasks train the model to "read between the lines," allowing it to guess the nature of an unknown entity from its usage in a sentence, even without a dictionary at hand.

Experimental Setup

The method was evaluated on two primary datasets:

  • Wizard of Wikipedia (WoW): A benchmark for knowledge-grounded dialogue, specifically the "Test Unseen" portion containing topics absent from the training set.
  • Reddit Trendings: A new dataset of 407 dialogues centered on 40 trending topics from 2021 that were largely absent from Wikipedia and the original Blender training data, simulating real-world information gaps.

Key Results

KE-Blender consistently outperformed standard Blender and other state-of-the-art knowledge-grounded models across several metrics:

  • Robustness: On the WoW "Test Unseen" set without knowledge available at inference, KE-Blender achieved a Perplexity (PPL) of 18.4 (vs. 19.9 for baseline) and an F1 score of 16.7 (vs. 12.9 for baseline).
  • Human Evaluation: Judges rated KE-Blender as more knowledgeable and coherent, particularly in settings where external knowledge was unavailable.
  • Graceful Degradation: A critical advantage of KE-Blender is its stability. As the availability of external knowledge decreases, KE-Blender's performance drops much more slowly than traditional knowledge-grounded models (KG-Blender).
Performance comparison showing KE-Blender's superior stability as knowledge availability decreases.

Conclusion

KE-Blender demonstrates that forcing a model to predict knowledge from context during training significantly improves its ability to handle unseen entities. By moving away from a total reliance on real-time retrieval, the model becomes more viable for "in-the-wild" applications where knowledge bases may be incomplete or ambiguous. Future work may involve integrating these auxiliary objectives into the large-scale pre-training phase to create even more robust foundation models.

Summary updated

Abstract

Although pre-training models have achieved great success in dialogue generation, their performance drops dramatically when the input contains an entity that does not appear in pre-training and fine-tuning datasets (unseen entity). To address this issue, existing methods leverage an external knowledge base to generate appropriate responses. In real-world scenario, the entity may not be included by the knowledge base or suffer from the precision of knowledge retrieval. To deal with this problem, instead of introducing knowledge base as the input, we force the model to learn a better semantic representation by predicting the information in the knowledge base, only based on the input context. Specifically, with the help of a knowledge base, we introduce two auxiliary training objectives: 1) Interpret Masked Word, which conjectures the meaning of the masked entity given the context; 2) Hypernym Generation, which predicts the hypernym of the entity based on the context. Experiment results on two dialogue corpus verify the effectiveness of our methods under both knowledge available and unavailable settings.
BibTeX
@inproceedings{cui2021knowledge,
  author = {Leyang Cui and Yu Wu and Shujie Liu and Yue Zhang},
  title = {{Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.179},
}
Figure