KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation

Wenhu Chen, Yu Su, Xifeng Yan, William Yang Wang

Abstract

Data-to-text generation has recently attracted substantial interests due to its wide applications. Existing methods have shown impressive performance on an array of tasks. However, they rely on a significant amount of labeled data for each task, which is costly to acquire and thus limits their application to new tasks and domains. In this paper, we propose to leverage pre-training and transfer learning to address this issue. We propose a knowledge-grounded pre-training (KGPT), which consists of two parts, 1) a general knowledge-grounded generation model to generate knowledge-enriched text. 2) a pre-training paradigm on a massive knowledge-grounded text corpus crawled from the web. The pre-trained model can be fine-tuned on various data-to-text generation tasks to generate task-specific text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under the fully-supervised setting, our model can achieve remarkable gains over the known baselines. Under zero-shot setting, our model without seeing any examples achieves over 30 ROUGE-L on WebNLG while all other baselines fail. Under the few-shot setting, our model only needs about one-fifteenth as many labeled examples to achieve the same level of performance as baseline models. These experiments consistently prove the strong generalization ability of our proposed framework.

Introduction

Data-to-text generation aims to transform structured data—such as RDF triples, tables, or sports statistics—into coherent natural language. While neural models have advanced significantly, they remain "data-hungry," requiring thousands of domain-specific labeled examples to perform reliably. Furthermore, standard language models often suffer from "hallucinations," generating fluent but factually incorrect text because they lack explicit grounding in structured knowledge.

Knowledge grounding between Wikipedia text and WikiData triples.

To address these challenges, we introduce KGPT (Knowledge-Grounded Pre-Training), a framework designed to learn the general mapping between data structures and human language. By pre-training on a massive corpus of data-text pairs, KGPT develops a robust understanding of how to verbalize diverse relationships, enabling high performance even in low-resource or zero-shot scenarios.

The KGTEXT Dataset

A core contribution of this work is the construction of KGTEXT, a large-scale pre-training corpus. Using distant supervision, we crawled Wikipedia for sentences containing hyperlinks and mapped these entities to WikiData to extract corresponding knowledge subgraphs (Subject-Predicate-Object triples). To ensure quality, we employed a lexical matching strategy, filtering out sentences that did not sufficiently overlap with the facts represented in the triples. This process resulted in a dataset of 1.8 million high-quality (knowledge subgraph, text) pairs.

Model Architecture

KGPT utilizes an encoder-decoder framework. We explored two primary encoder designs:

  1. Graph Encoder: A Graph Neural Network (GNN) that explicitly models the hierarchical relationships between entities, triples, and properties.
  2. Sequence Encoder: A Transformer-based model that processes a flattened version of the graph. To preserve structural information, we introduced structural embeddings—special tags for Entities, Triples, and Properties—that are added to the token embeddings.
The Sequence Encoder architecture with structural embeddings for entities, triples, and properties.

Our experiments revealed that the Sequence Encoder, despite its simplicity, performs nearly as well as the Graph Encoder, suggesting that "soft" structural hints are highly effective for text generation tasks.

Experimental Results

We evaluated KGPT across three benchmarks: WebNLG (RDF triples), E2ENLG (restaurant attributes), and WikiBio (biography infoboxes).

Fully-Supervised and Few-Shot Performance

KGPT achieved state-of-the-art results on WebNLG and E2E. Notably, in few-shot settings, KGPT demonstrated extreme efficiency, requiring approximately 15 times less labeled data than non-pre-trained baselines to achieve comparable performance. For instance, with only 40 training examples on the E2E task, KGPT maintained a BLEU score over 40, while baselines produced incoherent output.

Zero-Shot Generalization

In a zero-shot setting on WebNLG—where the model was tested without any task-specific fine-tuning—KGPT achieved a ROUGE-L score exceeding 30. In contrast, traditional models failed to generate meaningful text, highlighting KGPT's ability to generalize the logic of data verbalization to unseen domains.

Conclusion

KGPT demonstrates that large-scale knowledge-grounded pre-training is a powerful solution for data-to-text generation. By bridging the gap between structured data and natural language, the model reduces the dependency on expensive labeled datasets and improves generalization. Future work will focus on further reducing hallucinations and ensuring absolute faithfulness to the input data through more constrained decoding mechanisms.

Summary updated

Abstract

Data-to-text generation has recently attracted substantial interests due to its wide applications. Existing methods have shown impressive performance on an array of tasks. However, they rely on a significant amount of labeled data for each task, which is costly to acquire and thus limits their application to new tasks and domains. In this paper, we propose to leverage pre-training and transfer learning to address this issue. We propose a knowledge-grounded pre-training (KGPT), which consists of two parts, 1) a general knowledge-grounded generation model to generate knowledge-enriched text. 2) a pre-training paradigm on a massive knowledge-grounded text corpus crawled from the web. The pre-trained model can be fine-tuned on various data-to-text generation tasks to generate task-specific text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under the fully-supervised setting, our model can achieve remarkable gains over the known baselines. Under zero-shot setting, our model without seeing any examples achieves over 30 ROUGE-L on WebNLG while all other baselines fail. Under the few-shot setting, our model only needs about one-fifteenth as many labeled examples to achieve the same level of performance as baseline models. These experiments consistently prove the strong generalization ability of our proposed framework https://github.com/wenhuchen/KGPT.
BibTeX
@inproceedings{chen2020kgpt,
  author = {Wenhu Chen and Yu Su and Xifeng Yan and William Yang Wang},
  title = {{KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=u3gIaGC9yQw},
}
Figure