Describing a Knowledge Base

Qingyun Wang, Xiaoman Pan, Lifu Huang, Boliang Zhang, Zhiying Jiang, Heng Ji, Kevin Knight

Abstract

We aim to automatically generate natural language descriptions about an input structured knowledge base (KB). We build our generation framework based on a pointer network which can copy facts from the input KB, and add two attention mechanisms: (i) slot-aware attention to capture the association between a slot type and its corresponding slot value; and (ii) a new table position self-attention to capture the inter-dependencies among related slots. For evaluation, besides standard metrics including BLEU, METEOR, and ROUGE, we propose a KB reconstruction based metric by extracting a KB from the generation output and comparing it with the input KB. We also create a new data set which includes 106,216 pairs of structured KBs and their corresponding natural language descriptions for two distinct entity types. Experiments show that our approach significantly outperforms state-of-the-art methods. The reconstructed KB achieves 68.8% - 72.6% F-score.

Introduction

Structured Knowledge Bases (KBs), such as Wikidata or Wikipedia Infoboxes, are essential for storing precise, machine-readable data. However, for human consumption, this data is often difficult to parse quickly. Natural Language Generation (NLG) aims to bridge this gap by transforming structured facts into coherent, multi-sentence narratives.

Existing NLG models face two primary hurdles. First is semantic disconnection: standard models often lose the link between a "slot type" (e.g., Date of Birth) and its "slot value" (e.g., 1973), leading to "hallucinations" where correct data is assigned to the wrong categories. Second is structural complexity: KBs are not flat lists; facts are often interdependent (e.g., a player’s team, matches played, and goals scored are logically related). To address these, we introduce a novel neural framework and a large-scale dataset of 106,216 KB-description pairs derived from Wikipedia and Wikidata, focusing on "Person" and "Animal" entities.

Methodology

Our framework is built upon a Pointer-Generator Network, allowing the model to either generate words from a fixed vocabulary or "point" to and copy specific values directly from the input KB. This is critical for handling proper names and dates. We enhance this architecture with two specialized attention mechanisms:

  1. Slot-aware Attention: This mechanism computes attention for the slot type and slot value simultaneously. This ensures that when the model generates a word like "born," it specifically attends to the "Date of Birth" slot, significantly reducing the likelihood of mixing up different numerical values.
  2. Table Position Self-attention: Inspired by Transformer architectures, this captures inter-dependencies between different rows in the KB. By embedding both forward and backward row positions, the model learns to group related information into the same sentence and maintain a logical narrative flow.
Architecture of the proposed generation framework featuring slot-aware and table position attention.

Evaluation and Results

In KB-to-text tasks, fluency is secondary to factual accuracy. Standard metrics like BLEU, which measure word overlap, often fail to penalize factual swaps (e.g., swapping "matches played" with "goals scored"). Consequently, we propose a KB Reconstruction metric. In this process, we extract a KB from the generated text and compare it to the original input KB to measure how many facts were preserved.

The KB reconstruction evaluation process: generating text from a KB and then reconstructing the KB from that text.

Our proposed model showed dramatic improvements over standard Sequence-to-Sequence (Seq2seq) and basic Pointer models. On the "Person" dataset, BLEU scores increased from 11.3 (Seq2seq) to 23.2. More importantly, the model achieved an F1-score of 68.8% to 72.6% in the KB reconstruction task, a massive leap over the baseline Seq2seq model (~42%). Qualitatively, the model successfully grouped interdependent facts into single, coherent sentences, avoiding the disjointed or repetitive outputs common in baseline systems.

Discussion and Future Work

While our model significantly improves faithfulness, it still faces challenges with commonsense reasoning. For example, it may occasionally generate chronologically impossible sequences, such as a professional debut occurring before an entity's birth. Additionally, pronoun errors (gender detection) remain a minor issue.

Future research will focus on automating the KB reconstruction extraction phase to make the metric more scalable. We also plan to extend this framework to multilingual generation and cross-media applications, such as generating descriptions for images and videos based on underlying structured knowledge.

Summary updated

Abstract

We aim to automatically generate natural language descriptions about an input structured knowledge base (KB). We build our generation framework based on a pointer network which can copy facts from the input KB, and add two attention mechanisms: (i) slot-aware attention to capture the association between a slot type and its corresponding slot value; and (ii) a new \emph{table position self-attention} to capture the inter-dependencies among related slots. For evaluation, besides standard metrics including BLEU, METEOR, and ROUGE, we propose a KB reconstruction based metric by extracting a KB from the generation output and comparing it with the input KB. We also create a new data set which includes 106,216 pairs of structured KBs and their corresponding natural language descriptions for two distinct entity types. Experiments show that our approach significantly outperforms state-of-the-art methods. The reconstructed KB achieves 68.8% - 72.6% F-score.
BibTeX
@inproceedings{wang2018describing,
  author = {Qingyun Wang and Xiaoman Pan and Lifu Huang and Boliang Zhang and Zhiying Jiang and Heng Ji and Kevin Knight},
  title = {{Describing a Knowledge Base}},
  booktitle = {INLG},
  year = {2018},
  url = {https://openreview.net/forum?id=oxICGccWlK},
}
Figure