TL;DR
Generating natural, multi-sentence text directly from structured knowledge bases (KBs) often results in factual mix-ups and fragmented descriptions. This paper introduces two attention mechanisms—slot-aware attention and table position self-attention—built on top of a pointer-generator network to ensure facts stay paired with their correct attributes and interdependent facts are grouped into coherent sentences. Evaluated via standard n-gram metrics and a novel KB reconstruction evaluation, the model significantly improves factual fidelity.
What Problem Does This Paper Address?
Over half of the attributes stored in Wikipedia Infoboxes never make it into the corresponding Wikipedia article text. Bridging this gap requires Natural Language Generation (NLG) models that can take structured knowledge (a list of slot types and values) and transform it into fluent paragraphs.
Standard neural sequence-to-sequence approaches struggle with this task:
- Hallucination & Copy Errors: Pure seq2seq networks invent facts, while standard pointer networks copy values blindly, frequently assigning values to the wrong slot types (e.g., treating the number
22as a birth date rather than appearances in matches). - Disconnected Facts: Related table attributes (e.g., a football club, games played, and goals scored) are often scattered across disconnected sentences rather than presented cohesively.
- Dataset & Evaluation Limitations: Existing benchmarks (such as WikiBio) are mostly single-sentence descriptions. Standard automated evaluation metrics (BLEU, ROUGE) also fail to measure whether the underlying knowledge was accurately conveyed.
Methods
The authors adopt a sequence-to-sequence framework using bidirectional GRU encoders and an attention-equipped GRU decoder with a pointer-generator copy gate ($p_{\text{gen}}$) and coverage loss. They extend this with two innovations:
- Slot-Aware Attention: At each decoding step, the model computes attention over both slot types and slot values simultaneously, generating two separate context vectors ($L_s^*$ and $L_v^*$). This keeps values tightly bound to their semantic types during generation.
- Table Position Self-Attention: Structured triples often share latent structural relationships. By embedding forward ($r_i$) and backward ($\hat{r}_i$) row indices and computing self-attention across rows, the model captures dependencies between adjacent facts in the table. This allows the decoder to synthesize correlated slots into a single sentence.
- KB Reconstruction Metric: To directly measure factual fidelity, the authors manually reconstruct knowledge bases from generated descriptions, comparing extracted slots and interdependent rows against the input KB using Precision, Recall, and F1.
[Slot Type: Member of team] [Slot Value: Israel team]
\ /
(Slot-Aware + Table Position Attention)
|
v
"He was capped 22 times for the Israel women’s
national football team."
Datasets
The authors construct a new benchmark pairing Wikipedia dump text with Wikidata entity tables:
- Scale: 106,216 entity pairs across two domains:
- Person: 100,000 entities (76 filtered slot types; averaging 8.0 slots/table, 4.2 sentences, and 70.9 words per entity description).
- Animal: 6,216 entities (12 filtered slot types; averaging 3.2 slots/table, 2.5 sentences, and 42.2 words per entity description).
- Unlike prior single-sentence datasets (e.g., WikiBio), this dataset requires generating multi-sentence descriptions that cover extensive table records.
Key Insights and Ideas
- Pointer networks need schema guidance: While pointer networks prevent lexical hallucination, they lack type awareness. Explicitly conditioning generation on separate representations for slot types and slot values prevents "type-swapping" errors (e.g., mistaking match statistics for calendar dates).
- Table structure implies sentence structure: Interdependent attributes in tables (like an organization and a start date) naturally belong in the same sentence. Explicit positional self-attention effectively learns these latent groupings without requiring manual rule-based templates.
- Evaluate KBs as communication channels: Viewing natural language generation as transmitting structured data across a noisy channel makes reconstruction a natural evaluation criterion: can a downstream reader rebuild the original KB from the generated text?
Key Results
The proposed architecture (+Type & Position) outperformed standard Seq2seq and Pointer baselines across all metrics:
- Standard Metrics (BLEU / METEOR / ROUGE):
- Person: Reached 23.2 BLEU / 23.4 METEOR / 42.0 ROUGE (compared to Seq2seq’s 11.3 / 16.9 / 28.8 and Pointer's 17.2 / 21.1 / 37.4).
- Animal: Reached 14.8 BLEU / 17.2 METEOR / 45.0 ROUGE (compared to Seq2seq's 5.8 / 11.5 / 20.5).
- KB Reconstruction:
- Overall Slot Filling F1: Reached 68.8% on Person (up from 42.0% in Seq2seq) and 72.6% on Animal (up from 41.6% in Seq2seq).
- Inter-dependent Slot Filling F1: Achieved 69.7% on Person and 71.8% on Animal, demonstrating superior ability to associate linked attributes correctly.
Limitations and Future Work
- Commonsense & Temporal Failures: The language model component can introduce impossible facts (e.g., generating text claiming a soccer player debuted professionally before their birth year).
- Gender & Pronoun Agreement: The model occasionally generates incorrect pronouns due to missing gender signals in the structured representation.
- Generic Descriptions: For animal species, human writers elaborate with nuanced regional geography, whereas the model stays restricted to high-level facts present in the table.
- Ordering Assumptions: The current positional encoding assumes facts in the KB appear in temporal order. Future work aims to handle unordered slot tables and explore cross-lingual or cross-media (image/video) generation.
Summary updated