Presented at EMNLP 2022, this research addresses a significant bottleneck in Knowledge-Base-to-text (KB-to-text) generation: the trade-off between strict faithfulness and human-like informativeness. While traditional models are optimized to avoid "hallucinations," they often produce rigid, unhelpful text. This paper introduces the task of Informative Text Generation (ITG) and a novel architecture to solve it.
The Informative Text Generation Task
Standard KB-to-text tasks assume a one-to-one mapping between input data and output text. For instance, given the triple <Bill Gates, FounderOf, Microsoft>, a typical model outputs "Bill Gates founded Microsoft." However, human narratives naturally incorporate relevant background context—such as a person's birthplace or a company's headquarters—to provide a more comprehensive summary.
ITG requires models to "read between the lines" by generating text that includes relevant background information not explicitly provided in the input knowledge triples. This shift moves generation from mere translation of data to the creation of informative, human-like summaries.
The iMAG Framework
To tackle this task, the authors propose the information Memory Augmented Generator (iMAG). The architecture is designed to store and retrieve auxiliary knowledge during both training and inference.
Architecture and Training
The iMAG model consists of two primary components:
- Information Memory: A learnable memory matrix ($V$) that stores extra information encountered during training. To ensure the model actually uses this memory, the researchers employed a "triple-dropping" strategy: randomly removing input triples during training. This forces the model to retrieve the missing facts from its internal memory using the remaining entities as a query.
- Sequence-to-Sequence Network: An LSTM-based encoder-decoder with attention and a copy mechanism. It integrates the original input triples with retrieved memory vectors to generate the final narrative.
Handling Repetition
A common challenge in memory-augmented generation is content repetition. To mitigate this, the authors introduced the Repeating Sentence Penalty (RSP). By calculating the Longest Repeating Non-overlapping Substring, the model is penalized during training for generating redundant information, ensuring the output remains concise while being rich in content.
Key Results and Insights
The researchers evaluated iMAG on an adapted version of the WebNLG v2 dataset. Key findings include:
- Enhanced Richness: In human evaluations, iMAG+RSP scored significantly higher in "Richness" (7.02) compared to a standard Seq2Seq baseline (5.84).
- Substantial Output: The model generated significantly longer narratives, averaging 35.4 words per sample, compared to the baseline's 23.7 words.
- Zero-Triple Generation: Remarkably, iMAG demonstrated the ability to generate coherent entity biographies when provided with only an entity name and zero input triples, relying entirely on its internal "common sense" memory.
- Repetition Control: The RSP mechanism successfully reduced the repetition ratio (LRNSR) from 0.120 to 0.102 without compromising the quality of the generated information.
Conclusion
The iMAG framework represents a shift toward more functional and informative AI text generation. By treating memory as a repository for domain-specific common sense, the model moves beyond strict data-to-text mapping. While challenges remain in balancing plausibility with richness and scaling to open-domain settings, this work provides a robust foundation for generating narratives that are as helpful as they are faithful.
Summary updated