Abstract
Datasets for data-to-text generation typically focus either on multi-domain, single-sentence generation or on single-domain, long-form generation. In this work, we cast generating Wikipedia sections as a data-to-text generation task and create a large-scale dataset, WikiTableT, that pairs Wikipedia sections with their corresponding tabular data and various metadata. WikiTableT contains millions of instances, covering a broad range of topics, as well as a variety of flavors of generation tasks with different levels of flexibility. We benchmark several training and decoding strategies on WikiTableT. Our qualitative analysis shows that the best approaches can generate fluent and high quality texts but they struggle with coherence and factuality, showing the potential for our dataset to inspire future work on long-form generation.
Introduction
Data-to-text research—the task of transforming structured data into natural language—has traditionally been bifurcated. Existing datasets are either massive but limited to single-sentence generation (e.g., WIKIBIO), or they support long-form generation but are restricted to narrow domains like sports statistics (e.g., ROTOWIRE). This gap leaves a significant challenge for real-world applications, such as AI writing assistants, which must handle diverse topics and produce cohesive multi-sentence passages.
To address this, we introduce WikiTableT, a large-scale, multi-domain dataset that casts Wikipedia section writing as a data-to-text task. By pairing Wikipedia sections with structured data from infoboxes and Wikidata, we provide a benchmark that is both broad in scope and deep in complexity.
The WikiTableT Dataset
WikiTableT represents a significant scaling up of data-to-text resources. It contains approximately 1.5 million instances derived from over 887,000 unique Wikipedia articles. While 45% of the content focuses on biographies, the dataset covers a vast array of topics including films, music, geographical settlements, and albums.
The dataset structure includes:
- Article Data: Global context such as the document title and infobox attributes.
- Section Data: Specific facts relevant to a particular paragraph, identified through named entity recognition (NER) and hyperlink alignment.
Methodology
Generating long-form text from tables requires sophisticated modeling and decoding strategies. Our pipeline involves several key components:
Table Linearization
To process tabular data with Transformer models, we "flatten" tables into sequences using special tokens like boc (beginning of cell) and bov (beginning of value). Subscript indices are employed to maintain the association between attributes and their corresponding values.
Training Strategies
We experimented with three primary training enhancements:
- $\alpha$-entmax: A sparsity-promoting loss function designed to reduce "mushy" probability distributions.
- Copy Mechanism: Ensures that specific entities, such as names and dates, are accurately transferred from the table to the text.
- Cyclic Loss: A novel approach where a "backward" model attempts to reconstruct the original table from the generated text, incentivizing the generator to remain faithful to the input data.
Decoding Strategies
We compared standard greedy search and nucleus sampling against beam search with $n$-gram blocking. The latter is crucial for long-form generation as it prevents the model from entering repetitive loops by forbidding the repetition of specific trigrams.
Experimental Results
Our evaluation shows that the choice of decoding strategy and model size significantly impacts performance. The "Large" model using beam search with $n$-gram blocking achieved the best balance between fluency and faithfulness.
Key findings include:
- Faithfulness vs. Fluency: Models using $n$-gram blocking achieved a PAR-F1 score of 43.0 (measuring faithfulness) and a low repetition rate (REP score of 1.9).
- Human Evaluation: In blinded studies, the $n$-gram blocking model achieved scores of 4.2 for Grammaticality and 4.2 for Coherence on a 5-point scale, occasionally outperforming human-written reference texts.
- The Importance of Context: Including "Article Data" (global context) alongside "Section Data" significantly improved the relevance and quality of the generated text.
Discussion and Limitations
Despite these advances, challenges remain. Models still struggle with factuality and hallucinations, sometimes generating "common sense" contradictions (e.g., a character marrying before their birth date). Furthermore, the dataset is currently limited to the objective, encyclopedic tone of Wikipedia, which may not translate to more creative writing styles.
Future work should focus on improved entity modeling to help models maintain a consistent "memory" of different individuals or objects mentioned within the same table, thereby reducing confusion in complex narratives.
Conclusion
WikiTableT provides a robust foundation for advancing long-form, multi-domain data-to-text generation. By providing a massive, diverse dataset and benchmarking sophisticated training and decoding strategies, we highlight both the current capabilities of Transformer models and the significant hurdles that remain in achieving perfect factuality and coherence in automated writing.
Summary updated