Open Domain Event Text Generation

Zihao Fu, Lidong Bing, Wai Lam

Abstract

This paper introduces Open Domain Event Text Generation (ODETG), a task where models generate descriptive sentences based on a "skeleton" of entities (an entity chain) by retrieving supporting information from an external, trustworthy repository like Wikipedia. To solve this, we present the Wiki Augmented Generator (WAG) framework, which uses a novel "random drop" mechanism to prevent the model from overfitting on long, noisy retrieved text. Experimental results on the new WikiEvent dataset demonstrate that WAG significantly outperforms standard sequence-to-sequence baselines.

Introduction: The ODETG Task

Traditional Natural Language Generation (NLG) tasks—such as generating a restaurant review from a list of attributes—operate under a "closed-world" assumption: the model is expected to generate text using only the information provided in the input data.

The authors argue this is too rigid and doesn't reflect how humans actually write. For instance, if a writer wants to describe the "Punic War," they might start with a few key concepts (Punic War, Roman Republic, Ancient Carthage) and then research those entities to flesh out a complete, factual sentence. The ODETG task mimics this process, requiring models to bridge the gap between a sparse "entity chain" and a detailed description by looking up external knowledge.

The ODETG task: generating descriptive text from an entity chain by retrieving external knowledge.

The Wiki Augmented Generator (WAG)

To address the ODETG task, the authors propose the Wiki Augmented Generator (WAG), a framework consisting of three main components:

  1. Encoder: A standard Sequence-to-Sequence (S2S) encoder (multi-layer Bi-LSTM) that processes the input entity chain into a hidden representation.
  2. Retriever: Using the entities as queries, this component fetches relevant sentences from a pre-built Wikipedia repository via Elasticsearch.
  3. Decoder with Random Drop (RD): This is the core innovation. The decoder uses an attention mechanism to weigh both the encoded entity chain and the retrieved text.
The Wiki Augmented Generator (WAG) architecture, featuring the Encoder, Retriever, and Decoder with Random Drop.

The "Random Drop" Mechanism

Retrieved text can be extremely long and noisy. To prevent the model from simply memorizing or "overfitting" to specific retrieved segments during training, the authors implemented a Random Drop component. Unlike standard dropout (which zeroes out neurons), RD physically deletes random words from the retrieved sentences during training, forcing the model to be more robust and focus on the most salient information.

Experimental Setup

The authors introduced a new benchmark dataset called WikiEvent, consisting of 34,000 pairs of entity chains and description sentences derived from English Wikipedia. They filtered for articles related to events (using keywords like Battle, Revolution, Siege, Treaty) and extracted hyperlinked entities to form the chains. A standalone sentence repository was also built from Wikipedia for the retriever to query.

Key Results and Insights

The WAG model was evaluated against several baselines, including standard S2S models and versions that simply concatenated retrieved text to the input.

  • Performance: WAG outperformed all baselines across all metrics (BLEU, NIST, METEOR, ROUGE-L, and CIDEr). For example, WAG achieved a BLEU score of 0.238, compared to 0.200 for a standard S2S model using only entities.
  • The Signal-to-Noise Challenge: Retrieving external data is a double-edged sword. The authors found that retrieving just 1 or 2 sentences per entity yielded the best results; beyond that, the added noise began to degrade performance.
  • Ablation Study: Removing the Random Drop component (WAG w/o RD) led to a noticeable drop in performance on the test set, confirming that RD is essential for generalization.
  • Qualitative Success: Case studies showed the model successfully "filling in the blanks." For a chain like Tsardom of Russia — Constantinople — Ottoman Empire, the model was able to retrieve and include the specific date of a treaty (July 3, 1700) that was not present in the original entity chain.

Conclusion

The WAG framework demonstrates the power of augmenting generation with external knowledge while managing the resulting noise through structural innovations like Random Drop. Future work may involve expanding the repository depth beyond Wikipedia and using Reinforcement Learning to further refine how the model selects and utilizes retrieved text.

Summary updated

Abstract

Text generation tasks aim at generating human-readable text from different kinds of data. Normally, the generated text only contains the information included in the data and its application is thus restricted to some limited scenarios. In this paper, we extend the task to an open domain event text generation scenario with an entity chain as its skeleton. Specifically, given an entity chain containing several related event entities, the model should retrieve from a trustworthy repository (e.g. Wikipedia) the detailed information of these entities and generate a description text based on the retrieved sentences. We build a new dataset called WikiEvent1 that provides 34K pairs of entity chain and its corresponding description sentences. To solve the problem, we propose a wiki augmented generator framework that contains an encoder, a retriever, and a decoder. The encoder encodes the entity chain into a hidden space while the decoder decodes from the hidden space and generates description text. The retriever retrieves relevant text from a trustworthy repository which provides more information for generation. To alleviate the overfitting problem, we propose a novel random drop component that randomly deletes words from the retrieved sentences making our model more robust for handling long input sentences. We apply the proposed model on the WikiEvent dataset and compare it with a few baselines. The experimental results show that our carefully-designed architecture does help generate better event text, and extensive analysis further uncovers the characteristics of the proposed task.
BibTeX
@inproceedings{fu2020open,
  author = {Zihao Fu and Lidong Bing and Wai Lam},
  title = {{Open Domain Event Text Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6278},
}
Figure