DESCGEN: A Distantly Supervised Datasetfor Generating Entity Descriptions

Weijia Shi, Mandar Joshi, Luke Zettlemoyer

TL;DR

This paper introduces DESCGEN, a new dataset for abstractive entity description generation, which aims to better capture the challenge of describing new and emerging entities. They also propose a two-stage baseline method and demonstrate a significant gap between current models and human performance, highlighting the need for further research.

What Problem Does This Paper Address?

Entity descriptions are crucial for various NLP tasks, acting as a knowledge source for tasks like question answering and dialogue generation. However, manually creating these descriptions is labor-intensive, especially for the ever-growing number of new and "long-tail" entities that lack extensive existing documentation. Existing methods often rely on structured knowledge graphs, which can be incomplete or outdated, or on highly extractive summaries from a few curated sources. This paper tackles the challenge of generating abstractive entity descriptions from a diverse set of potentially scattered documents, mimicking how humans might synthesize information about a new entity.

Methods

The authors propose a two-stage approach for generating entity descriptions:

  1. Extractive Stage: This stage identifies representative sentences from the source documents that are relevant to the entity. They compare several methods, including a simple "Trivial" approach (concatenating sentences mentioning the entity with surrounding context) and a BERT-based classifier trained to predict sentence relevance.
  2. Abstractive Stage: In this stage, a neural generation model fuses the selected sentences into a concise, abstractive description. They fine-tune pre-trained language models like BART, T5, and MARGE for this task.

Datasets

The core contribution is the DESCGEN dataset, which comprises:

  • 37K entity descriptions paired with an average of nine source documents.
  • Descriptions are sourced from Wikipedia and Fandom, with a focus on including entities not present in Wikipedia to better represent emerging entities.
  • A human-annotated subset of 1,000 examples for more reliable evaluation.

This dataset is designed to be more abstractive than existing benchmarks like WikiSum, which tend to be more extractive due to reliance on curated citation sources.

Key Insights and Ideas

  • Abstractiveness is Key for Emerging Entities: The authors argue that for new or emerging entities, information is often scattered across diverse sources with varied styles. This necessitates abstractive summarization rather than simply extracting sentences, making DESCGEN a more realistic benchmark.
  • Distant Supervision is Challenging but Necessary: While distant supervision (using heuristically collected documents) is not guaranteed to contain all facts, it's crucial for scaling to a large number of entities, especially those without readily available curated sources. The paper explores the quality of this supervision.
  • Significant Gap Between Models and Humans: The results show a substantial performance gap between state-of-the-art models and human performance, indicating that current models struggle with the abstractive nature of the task and capturing nuanced entity knowledge.
  • Domain and Source Diversity Matters: Performance varies across different entity domains and sources (Wikipedia vs. Fandom), suggesting that models need to generalize better to less popular domains and diverse writing styles.

Key Results

  • The DESCGEN dataset contains 37K entity-description pairs, with an average of nine source documents per entity.
  • The best baseline model (BART) achieved a ROUGE-L score of 28.2% on the distant supervision test set, while human performance reached 48.1%. This highlights a significant gap of 19.9%.
  • On the verified (human-annotated) test set, the gap was even larger, with the best model achieving 26.6% ROUGE-L and human performance at 48.1%.
  • Manual evaluation showed that while models like BART generated fluent descriptions, they struggled with faithfulness, sometimes synthesizing information not directly inferable from the source text or copying irrelevant details.
  • Models trained on entity names alone performed worse than those using source documents, suggesting that while pre-trained models might store some factual knowledge, source documents are essential for accurate description generation.

Limitations and Future Work

  • Extractive Content Selection Difficulty: The paper acknowledges that extractive content selection remains a challenge, as evidenced by the gap between BERT extraction and a heuristic "Cheating" model.
  • Abstractive Generation Challenges: Models struggle with abstractive generation, particularly in ensuring faithfulness to the source text and avoiding the inclusion of irrelevant or incorrect information.
  • Future Work: The authors suggest future research directions focusing on:
  • Modeling entity knowledge from contexts: Developing better ways to leverage contextual information for richer entity understanding.
  • Improving extractive content selection: Enhancing the ability to identify the most salient information.
  • Enhancing abstractive generation: Focusing on faithfulness and accuracy in synthesized descriptions.
  • Incorporating entity knowledge into downstream tasks: Applying these generated descriptions to improve performance in other NLP applications.

Summary updated

BibTeX
@inproceedings{shi2021descgen,
  author = {Weijia Shi and Mandar Joshi and Luke Zettlemoyer},
  title = {{DESCGEN: A Distantly Supervised Datasetfor Generating Entity Descriptions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Ss6lAi5P0P},
}
Figure