Introduction
Data-to-text generation—the task of converting structured data into natural language—is a cornerstone of automated content creation. Historically, research in this field has been bifurcated. Datasets like WikiBio offer broad topical diversity but are limited to single-sentence generation. Conversely, datasets like RotoWire provide long-form narratives but are confined to narrow domains, such as basketball game summaries. This leaves a significant gap for benchmarks that are both large-scale and multi-domain while requiring long-form output.
Researchers from TTIC have introduced WikiTableT, a massive dataset of 1.5 million instances designed to bridge this gap. By pairing Wikipedia article sections with structured data from infoboxes, Wikidata, and named entities, WikiTableT provides a diverse benchmark that challenges models to maintain long-range coherence and factual consistency across a wide array of topics.
The WikiTableT Dataset
The primary contribution of this work is the WikiTableT dataset, which significantly expands the scale and diversity of previous long-form benchmarks. While RotoWire contains approximately 5,000 instances, WikiTableT offers ~1.5 million, covering topics ranging from biographies and films to music and human settlements.
The input for each instance is a combination of three data types:
- Article Data: Global information typically found in infoboxes.
- Section Data: Local entities and hyperlinks specific to the section being generated.
- Title Data: The hierarchical structure of the article, including document and section titles.
The dataset's diversity is a key feature, ensuring that models must generalize across many linguistic styles and factual domains. As shown in the category distribution, while human biographies form the largest share, the dataset includes significant portions of films, music, and geographical data.
Methodology
To benchmark WikiTableT, the authors utilized standard Transformer architectures (Base and Large) with several adaptations for tabular input:
- Table Linearization: Tables are flattened into sequences using special tokens:
<boc>(beginning of cell),<bov>(beginning of value), and<eoc>(end of cell). To prevent the model from over-relying on row order, positional embeddings are restarted for each record. - Training Enhancements: The authors employed $\alpha$-entmax loss to reduce word-level repetition and a Copy Mechanism to handle rare entities like specific dates or names. Additionally, a Cyclic Loss was implemented, where a backward model reconstructs the original data from the generated text to encourage faithfulness.
- Decoding Strategies: Standard beam search often leads to repetitive loops in long-form generation. The authors found that $n$-gram blocking was essential for maintaining output quality.
Key Insights and Results
The authors' evaluation revealed a "flexibility spectrum" within the dataset. Models performed well on highly constrained tasks where the table contained all necessary facts. However, in more "open-ended" tasks—such as fictional character biographies—models frequently hallucinated details to fill gaps in the structured data.
Fluency vs. Factuality
Human evaluations indicated that model-generated text often matched human-written Wikipedia sections in terms of Grammaticality (4.2/5) and Coherence (4.2/5). Despite this fluency, models struggled with Support (factuality). Common errors included chronological impossibilities (e.g., a person being married before birth) or confusing the sequence of events.
Performance Metrics
The "Large" Transformer model using beam search with $n$-gram blocking achieved a BLEU score of 19.3 and a PARENT-F1 score of 43.0. The latter is particularly important as it measures faithfulness to the input table. The results suggest that while current models are surprisingly fluent, maintaining a logical "thread" across multiple sentences while staying strictly faithful to complex input remains a significant challenge.
Conclusion and Future Work
WikiTableT establishes a new frontier for long-form data-to-text generation. The findings highlight that the primary bottleneck is no longer fluency, but logical coherence and factual integrity. Future research should focus on reducing "impossible" factual errors through better world-knowledge integration and developing explicit modules to ensure generated text remains grounded in the input data.
Summary updated