Triplet-Free Knowledge-Guided Response Generation

Dongming Li, Jianfeng Liu, Baoyuan WangACL 2023 · OpenReview

Abstract

Generating vivid and informative responses (e.g., comments for social posts and utterances for dialogues) is challenging without giving relevant knowledge. Prior works focus on constructing the "latent" knowledge first and then learning how to "ground" it based on pseudo (context, knowledge, response) triplets. However, the retrieval between real responses and their latent knowledge is difficult in nature. In this paper, instead of focusing on how to ground knowledge given the responses, we take a different perspective to optimize the final responses for given guided knowledge directly. This allows us to re-formulate the entire problem in a simplified yet more scalable way. Specifically, we pretrain a response language model (LM) to measure the relevance and consistency between any context and response, then use search engines to collect the top-ranked passages to serve as the guiding knowledge without explicitly optimizing the "best" latent knowledge that corresponds to a given response. The final response generation model is trained through reinforcement learning by taking both the response LM prior and knowledge-injection rate as rewards. For better evaluations, we construct a new Chinese benchmark, "IceKC", using fresh multimodal online social posts. Both automatic evaluations and human evaluations show our zero-resource approach performs significantly better than prior works.

Introduction

Generating responses that are both engaging and factually grounded is a major hurdle in conversational AI. Most current models rely on "knowledge grounding," which typically requires training on triplets: a conversation context ($C$), a specific piece of knowledge ($K$), and a target response ($R$).

The bottleneck is that high-quality $(C, K, R)$ triplets are incredibly hard to find or expensive to build. In the real world, the "knowledge" behind a comment is often scattered across the web or only loosely related. Previous "zero-resource" attempts tried to guess the best latent knowledge for a response, but these "pseudo-triplets" are often noisy and unreliable, leading to models that either ignore the knowledge or produce nonsensical text.

The proposed framework shifts from optimizing latent knowledge to directly optimizing the response based on retrieved knowledge.

Methodology

The authors propose a shift in perspective: instead of trying to find the "perfect" knowledge for a response, they optimize the response directly for whatever knowledge is provided. They formulate this as a reinforcement learning (RL) problem where the generator is trained to maximize a total reward from two "critics":

  1. Consistency Reward ($r_c$): A pretrained language model (LM) acts as a critic to ensure the response is relevant to the conversation context and sounds natural. They use an "unlikelihood objective" during pretraining to discourage the model from repeating itself or being too generic.
  2. Knowledge Injection Reward ($r_k$): This non-parametric critic measures how much information from the provided knowledge actually made it into the response. Currently, this is calculated using a simple but effective $n$-gram matching score (BLEU-2) between the response and the knowledge snippet.

To tie it all together, they use Soft Q-Learning to optimize the generator. This allows the model to learn from unpaired data—essentially practicing how to weave facts into a conversation without needing a "ground truth" answer for every specific fact-context pair.

The RL-based training architecture utilizes a Transformer generator and two distinct reward signals for consistency and knowledge injection.

Experimental Setup

The study evaluates the approach across three major benchmarks:

  • Wizard of Wikipedia (WoW): A standard English dataset for knowledge-grounded dialogue.
  • Wizard of Internet (WizInt): An English dataset where knowledge is retrieved from the open web.
  • IceKC (New): The authors introduced this new Chinese benchmark. It consists of nearly 1,000 instances derived from Weibo (social media) posts. Unlike previous datasets, IceKC is multimodal, containing both text and images, and features "fresh" topics from 2022 to ensure the models can't rely solely on their internal training data.

Key Results and Insights

The proposed model consistently outperformed triplet-based baselines (like ZRKGC and KAT-TSLF) across almost all metrics:

  • Knowledge Integration: On the WoW "test unseen" set, the model achieved a Knowledge F1 (KF1) of 0.494, significantly higher than the 0.152 to 0.421 range of the baselines.
  • Human Evaluation: In double-blind human trials on WizInt samples, the proposed model scored higher in Knowledge Relevance (1.4 vs. 0.6-1.3) and Fluency (1.9 vs. 1.3-1.8) compared to other zero-resource methods.
  • Zero-Shot Superiority: The approach even beat massive pretrained models like PANGU-$\alpha$ (2.6B parameters) and EVA (2.8B parameters) in its ability to actually incorporate external facts, despite being a much smaller model (220M parameters).

Key Insights: * The "Triplet-Free" Advantage: By removing the requirement for explicit $(C, K, R)$ alignment, the model becomes much more scalable. It can learn from any conversation and any relevant search result. * Decoupling Consistency and Facts: The two-critic approach allows the model to balance two often-conflicting goals: being a good conversationalist (Consistency) and being a subject matter expert (Knowledge Injection). * Multimodal Synergy: On the IceKC benchmark, the model performed better when it had access to images from the original social media posts, suggesting that visual context helps the AI better understand how to apply retrieved knowledge.

Conclusion

While the results are promising, several areas for improvement remain. Like many RL-based models, the generator sometimes finds "shortcuts" to fool the critics, leading to slightly repetitive responses. Additionally, the reliance on $n$-gram matching for the knowledge reward may limit creative paraphrasing. Future work will explore more sophisticated reward functions and deeper integration of multimodal information to further enhance the richness of generated responses.

Summary updated

Abstract

Generating vivid and informative responses (e.g., comments for social posts and utterances for dialogues) is challenging without giving relevant knowledge. Prior works focus on constructing the ”latent” knowledge first and then learning how to ”ground” it based on pseudo (context, knowledge, response) triplets. However, the retrieval between real responses and their latent knowledge is difficult in nature. In this paper, instead of focusing on how to ground knowledge given the responses, we take a different perspective to optimize the final responses for given guided knowledge directly. This allows us to re-formulate the entire problem in a simplified yet more scalable way. Specifically, we pretrain a response language model (LM) to measure the relevance and consistency between any context and response, then use search engines to collect the top-ranked passages to serve as the guiding knowledge without explicitly optimizing the ‘‘best” latent knowledge that corresponds to a given response. The final response generation model is trained through reinforcement learning by taking both the response LM prior and knowledge-injection rate as rewards. For better evaluations, we construct a new Chinese benchmark, ”IceKC”, using fresh multimodal online social posts. Both automatic evaluations and human evaluations show our zero-resource approach performs significantly better than prior works.
BibTeX
@inproceedings{li2023triplet,
  author = {Dongming Li and Jianfeng Liu and Baoyuan Wang},
  title = {{Triplet-Free Knowledge-Guided Response Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=RTRGVj2UNX},
}
Figure