Abstract
Short textual descriptions of entities provide summaries of their key attributes and have been shown to be useful sources of background knowledge for tasks such as entity linking and question answering. However, generating entity descriptions, especially for new and long-tail entities, can be challenging since relevant information is often scattered across multiple sources with varied content and style. We introduce DESCGEN: given mentions spread over multiple documents, the goal is to generate an entity summary description. DESCGEN consists of 37K entity descriptions from Wikipedia and Fandom, each paired with nine evidence documents on average. The documents were collected using a combination of entity linking and hyperlinks to the Wikipedia and Fandom entity pages, which together provide high-quality distant supervision. The resulting summaries are more abstractive than those found in existing datasets and provide a better proxy for the challenge of describing new and emerging entities. We also propose a two-stage extract-then-generate baseline and show that there exists a large gap (19.9% in ROUGE-L) between state-of-the-art models and human performance, suggesting that the data will support significant future work.
Introduction
In the field of Natural Language Processing (NLP), concise entity descriptions—brief "bios" of people, places, or fictional characters—are vital for downstream tasks like question answering and entity linking. While manually curating these descriptions is labor-intensive, automating the process is fraught with challenges. Existing datasets, such as WikiSum, often rely on high-quality citations already present in Wikipedia, making the task "too easy" as models can often succeed by simply extracting and reordering existing sentences.
Furthermore, these datasets often ignore "long-tail" or emerging entities—newly famous individuals or niche topics that lack established Wikipedia entries. To address these limitations, we introduce DESCGEN, a dataset designed to push the boundaries of abstractive summarization by focusing on diverse, multi-source information synthesis.
The DESCGEN Dataset
DESCGEN contains 37,000 entity descriptions. To build this resource, we employed a distant supervision approach: using existing descriptions from Wikipedia and Fandom as ground truth, we automatically identified relevant web documents through hyperlinks and entity linkers.
A key feature of DESCGEN is its diversity. Approximately 36% of the entities are sourced from Fandom, a platform for enthusiast-driven wikis. This ensures coverage of domains like gaming, movies, and television series that are often absent from standard Wikipedia-based datasets.
Quantitatively, DESCGEN is significantly more abstractive than older benchmarks like CNN/Daily Mail. It features lower "density" scores, indicating that models cannot rely on verbatim copying and must instead synthesize information into novel phrasing.
Methodology: The Extract-then-Generate Pipeline
To establish a baseline for DESCGEN, we proposed a two-stage "Extract-then-Generate" pipeline:
- Extractive Stage: A BERT-based classifier scans multiple source documents to select the most relevant sentences.
- Abstractive Stage: A sequence-to-sequence model (such as BART or T5) takes the selected sentences and fuses them into a final, coherent description.
This approach mimics how a human might research a topic across multiple sources before writing a summary.
Key Insights: The Faithfulness Gap
Our results highlight a massive performance gap between current state-of-the-art models and human writers. While the BERT+BART combination achieved a ROUGE-L score of 28.2 on our verified test set, human performance reached 48.1—a 19.9% gap.
Analysis suggests several reasons for this disparity:
- The "Copy-Paste" Trap: Current models often act as "intelligent scissors," cutting and joining phrases rather than truly paraphrasing. They struggle to generate novel words that capture the essence of the source material.
- Pre-training Bias: Models like BART and T5 are pre-trained on Wikipedia. Consequently, they perform better on Wikipedia entities they have "memorized" but struggle with the long-tail Fandom entities where they must rely solely on the provided evidence.
- Faithfulness and Hallucinations: Models frequently suffer from "hallucinations"—misattributing facts or inventing details. For instance, a model might confuse a subject with a relative mentioned in the same document.
Conclusion and Future Work
DESCGEN reveals that while modern models are fluent, they lack the synthesis and factual rigor required for high-quality entity description. Future work must focus on improving factuality and content selection to ensure models remain faithful to their sources. By providing a challenging, abstractive, and diverse benchmark, DESCGEN paves the way for more robust and reliable summarization systems.
Summary updated