Injecting Entity Types into Entity-Guided Text Generation

Xiangyu Dong, Wenhao Yu, Chenguang Zhu, Meng Jiang

Introduction

Generating coherent news stories from a sparse list of entities—such as names and locations—presents a significant "sparse-to-dense" challenge for natural language generation models. In Entity-Guided Text Generation, a model must hallucinate the relationships and "bridge" words (verbs, prepositions, etc.) to connect entities like "Dick Cheney," "Afghanistan," and "Monday" into a natural sentence.

Existing methods typically rely on class concatenation, where an entity name and its type are combined into a single embedding and fed into a standard Seq2Seq model. However, this provides a weak signal, often failing to deeply model the relationship between a name and its type. This frequently results in incomplete or illogical sentences. To address this, the authors propose InjType, a model that explicitly injects entity type information directly into the decoding process.

Methods: The InjType Architecture

InjType moves beyond simple concatenation by employing a multi-step "injection" strategy during decoding. This approach forces the model's hidden states to learn the functional role of an entity before generating its specific surface name.

The InjType architecture illustrating the four-step decoding process: (1) Entity Indicator Prediction, (2) Type Injection, (3) Mention Prediction, and (4) Type-Enhanced NLU.

The decoding process follows four specific steps:

  1. Entity Indicator Prediction: The decoder first decides whether to generate a standard word or an entity. If an entity is chosen, it generates a special <Ent> token.
  2. Type Injection: Upon triggering an <Ent> token, the model injects the specific embedding of that entity's type (e.g., COUNTRY) into the decoder's hidden state.
  3. Mention Prediction: A specialized mention predictor uses this type-aware hidden state to predict the actual surface name (e.g., "US").
  4. Type-Enhanced NLU (Auxiliary Task): During training, a secondary Natural Language Understanding task requires the model to predict entity mentions from ground-truth context words in reverse. This ensures the decoder's hidden states carry rich information about how context relates to specific entity types.

Experimental Setup

The researchers evaluated InjType on two primary news datasets:

  • GIGAWORDS-6K: 7,903 articles from various news publishers (1994–2010).
  • NYT-8K: 8,371 article summaries from the New York Times (1987–2007).

Using Stanford CoreNLP, the authors extracted 14 entity types, including COUNTRY, PERSON, ORGANIZATION, WEEKDAY, and CITY.

Key Results and Insights

InjType demonstrated substantial improvements over strong baselines, including Seq2Seq with attention, CopyNet, and large-scale pre-trained models like GPT-2 and UniLM.

Quantitative performance of InjType compared to baselines across GIGAWORDS and NYT datasets.

Quantitative Performance

On the GIGAWORDS dataset, InjType achieved a BLEU-4 score of 18.55, outperforming UniLM (17.66) and SeqAttn (16.17). ROUGE-L scores also saw a significant increase, rising from approximately 36 in baseline models to 41.16. Ablation studies confirmed that the Mention Predictor (MP) is the most critical component; its removal led to a massive performance drop.

Human Evaluation and Qualitative Insights

Human evaluations via Amazon Mechanical Turk rated InjType significantly higher in Coherence (48% win rate vs. SeqAttn) and Informativeness (50.6% win rate). A key insight is that entity types serve as a better "bridge" for predicting surrounding context than specific names. For instance, knowing a PERSON is the subject helps the model accurately select verbs like "said" or "visited."

Furthermore, InjType—a relatively compact model—outperformed much larger models like GPT-2. This suggests that for constrained, entity-heavy generation, specific architectural inductive biases like type injection can be more effective than raw scaling.

Conclusion and Future Work

While InjType provides a powerful mechanism for entity-guided generation, it currently relies on a predefined entity list and a limited set of 14 broad types. Future research could explore more granular type systems (e.g., distinguishing between Politician and Athlete) or integrating this injection mechanism into massive pre-trained transformers to further enhance performance.

Summary updated

Abstract

Recent successes in deep generative modeling have led to significant advances in natural language generation (NLG). Incorporating entities into neural generation models has demonstrated great improvements by assisting to infer the summary topic and to generate coherent content. To enhance the role of entity in NLG, in this paper, we aim to model the entity type in the decoding phase to generate contextual words accurately. We develop a novel NLG model to produce a target sequence based on a given list of entities. Our model has a multi-step decoder that injects the entity types into the process of entity mention generation. Experiments on two public news datasets demonstrate type injection performs better than existing type embedding concatenation baselines.
BibTeX
@inproceedings{dong2021injecting,
  author = {Xiangyu Dong and Wenhao Yu and Chenguang Zhu and Meng Jiang},
  title = {{Injecting Entity Types into Entity-Guided Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.56},
}
Figure