Injecting Entity Types into Entity-Guided Text Generation

Xiangyu Dong, Wenhao Yu, Chenguang Zhu, Meng Jiang

Abstract

Recent successes in deep generative modeling have led to significant advances in natural language generation (NLG). Incorporating entities into neural generation models has demonstrated great improvements by assisting to infer the summary topic and to generate coherent content. To enhance the role of entity in NLG, in this paper, we aim to model the entity type in the decoding phase to generate contextual words accurately. We develop a novel NLG model to produce a target sequence based on a given list of entities. Our model has a multi-step decoder that injects the entity types into the process of entity mention generation. Experiments on two public news datasets demonstrate type injection performs better than existing type embedding concatenation baselines.

Introduction

Generating a full news story or summary from a sparse list of names and categories—a task known as words-to-text generation—is notoriously difficult. Because the input is minimal (e.g., "US: Country, Dick Cheney: Person"), standard models often struggle to find the right "connective tissue" or contextual words to link these entities logically.

Existing approaches usually just concatenate the entity name and its type at the start of the process. However, this often leads to the "type" signal getting lost in the noise, resulting in models that fail to mention all required entities or produce "hallucinations" where the relationship between people and places is nonsensical. The researchers developed InjType, a novel NLG model that explicitly "injects" entity types into the decoding process as first-class citizens, generating more accurate and contextually relevant text.

Methodology

The core innovation of InjType is its multi-step decoder. Instead of predicting the next word in a single pass, the decoder follows a four-step process for every entity:

  1. Entity Indicator Prediction: The model first decides if the next token should be a regular word or an entity (marked by a special <Ent> tag).
  2. Type Injection: If it is an entity, the model "injects" the specific type embedding (e.g., the concept of a "PERSON") directly into the current hidden state of the neural network.
  3. Mention Prediction: A specialized "mention predictor" uses this type-aware hidden state to pick the specific name (e.g., "Dick Cheney") from the input list.
  4. Type-Enhanced NLU (Auxiliary Task): During training, the model performs a "Natural Language Understanding" task using a bidirectional GRU to guess which entity belongs in a given context. This forces the decoder to learn which contextual words (like "visited" or "spoke with") are best suited for specific entity types.
The InjType architecture featuring the multi-step decoder and type injection process.

By separating the role of the entity (the type) from its identity (the mention), the model develops a stronger grasp of how to build sentences around them. This approach treats types as anchors; knowing a "PERSON" is coming allows the model to prepare verbs like "said" or "met," whereas a "CITY" might trigger "arrived in."

Experimental Setup

The authors evaluated InjType on two large-scale news datasets:

  • GIGAWORD-6K: Approximately 6,000 articles from various news publishers (1994–2010).
  • NYT-8K: Approximately 8,000 article summaries from the New York Times (1987–2007).

Both datasets were processed using Stanford CoreNLP to identify 14 specific entity types, including PERSON, COUNTRY, ORGANIZATION, WEEKDAY, and CITY.

Key Results

InjType significantly outperformed competitive baselines, including large-scale models like GPT-2 and UniLM, despite having far fewer parameters and no massive pre-training.

Quantitative comparison of InjType against various baselines on GIGAWORD and NYT datasets.

On the GIGAWORD dataset, InjType achieved a BLEU-4 score of 18.55, compared to SeqAttn's 16.17 and UniLM's 17.66. It showed similar dominance in ROUGE-2 and ROUGE-L metrics across both datasets. Qualitatively, human evaluators found that InjType's outputs were significantly more coherent (winning 48% of head-to-head matchups) and informative (winning 50.6%) than standard attention-based models. Ablation studies confirmed that the "Mention Predictor" was the most critical component of the architecture.

Conclusion and Future Work

InjType demonstrates that explicit type injection is a powerful mechanism for entity-guided generation. While the model performs exceptionally well, future work could involve scaling the architecture with large-scale pre-training or expanding the entity diversity beyond the 14 standard NER types. Additionally, integrating external knowledge bases could ensure that the generated relationships between entities are not just linguistically plausible, but factually accurate.

Summary updated

Abstract

Recent successes in deep generative modeling have led to significant advances in natural language generation (NLG). Incorporating entities into neural generation models has demonstrated great improvements by assisting to infer the summary topic and to generate coherent content. To enhance the role of entity in NLG, in this paper, we aim to model the entity type in the decoding phase to generate contextual words accurately. We develop a novel NLG model to produce a target sequence based on a given list of entities. Our model has a multi-step decoder that injects the entity types into the process of entity mention generation. Experiments on two public news datasets demonstrate type injection performs better than existing type embedding concatenation baselines.
BibTeX
@inproceedings{dong2021injecting,
  author = {Xiangyu Dong and Wenhao Yu and Chenguang Zhu and Meng Jiang},
  title = {{Injecting Entity Types into Entity-Guided Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.56},
}
Figure