Abstract
Data-to-text generation has recently attracted substantial interests due to its wide applications. Existing methods have shown impressive performance on an array of tasks. However, they rely on a significant amount of labeled data for each task, which is costly to acquire and thus limits their application to new tasks and domains. In this paper, we propose to leverage pre-training and transfer learning to address this issue. We propose a knowledge-grounded pre-training (KGPT), which consists of two parts, 1) a general knowledge-grounded generation model to generate knowledge-enriched text. 2) a pre-training paradigm on a massive knowledge-grounded text corpus crawled from the web. The pre-trained model can be fine-tuned on various data-to-text generation tasks to generate task-specific text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under the fully-supervised setting, our model can achieve remarkable gains over the known baselines. Under zero-shot setting, our model without seeing any examples achieves over 30 ROUGE-L on WebNLG while all other baselines fail. Under the few-shot setting, our model only needs about one-fifteenth as many labeled examples to achieve the same level of performance as baseline models. These experiments consistently prove the strong generalization ability of our proposed framework.
Introduction
Data-to-text generation—the task of transforming structured data such as RDF triples or tables into fluent natural language—is essential for applications ranging from automated weather reports to virtual assistants. However, traditional neural models are bottlenecked by data scarcity, requiring vast amounts of human-annotated examples for every new domain. While general-purpose language models like GPT-2 generate fluent text, they often "hallucinate" facts because they are not inherently designed to ingest structured data.
KGPT addresses this by introducing a framework that is "knowledge-grounded" from the start, learning the direct mapping between structured knowledge subgraphs and narrative text.
Methodology: KGTEXT and KGPT
The authors propose a two-stage approach involving a novel dataset and a specialized model architecture.
KGTEXT Dataset Construction
To bypass the high cost of manual labeling, the researchers utilized distant supervision to create KGTEXT. By crawling Wikipedia for sentences containing hyperlinks and linking those entities to Wikidata triples, they automatically generated 1.8 million knowledge-subgraph-to-text pairs. To ensure high quality, a lexical-based selection strategy was employed to filter out noisy pairs, retaining only sentences with strong semantic overlap with their associated triples.
Model Architecture
KGPT utilizes an encoder-decoder structure. While the authors explored Graph Attention Networks (GAT), they found that a Sequence Encoder based on the Transformer architecture performed comparably well when augmented with specialized structural information.
As shown in the architecture diagram, the model uses a hierarchy of embeddings—Token, Entity, Triple, and Property—to "softly" inject graph structure into the vector space. This allows the model to understand the role of each token (e.g., subject, predicate, or object) within a specific knowledge triple.
Key Results and Generalization
The effectiveness of KGPT was evaluated across several benchmarks, including WebNLG, E2ENLG, and WikiBio.
- State-of-the-Art Performance: In fully supervised settings, KGPT outperformed existing baselines, achieving 64.11 BLEU on WebNLG and 68.05 BLEU on E2E.
- Zero-Shot Capabilities: Remarkably, KGPT achieved a ROUGE-L score of over 30 on WebNLG without seeing any task-specific training examples. In contrast, traditional models failed completely in this setting.
- Data Efficiency: KGPT demonstrates a 15x reduction in annotation costs. It can match the performance of standard supervised baselines using only ~6% of the labeled training data.
- Factual Fidelity: Human evaluation confirmed that KGPT is significantly less prone to hallucinations than non-pre-trained models, consistently producing more accurate descriptions.
Conclusion
KGPT demonstrates that pre-training on a massive, automatically constructed knowledge-grounded corpus significantly enhances a model's ability to generate faithful and fluent text from structured data. By bridging the gap between knowledge graphs and natural language, KGPT offers a path toward more robust and data-efficient generation systems.
Summary updated