Table-to-Text Natural Language Generation with Unseen Schemas

Tianyu Liu, Wei Wei, William Yang WangarXiv 2019 · arXiv

Abstract

Traditional table-to-text generation models often fail when encountering "unseen schemas"—attribute types not present during training. We present AlignNet, a framework that explicitly aligns novel attributes to known ones from a support set using a Schema Aligner module. By treating unseen attributes as semantic paraphrases of known ones, AlignNet significantly improves generalization in open-world, low-resource scenarios, achieving up to a 33.3% relative improvement in BLEU-4 scores on small datasets.

Introduction

Natural Language Generation (NLG) from structured data typically operates in a "closed-world" assumption. Models are trained on specific domains, such as weather reports or restaurant reviews, where the attributes (e.g., "temperature" or "cuisine") are fixed. However, real-world data is infinite and diverse. A model trained on baseball statistics may struggle with cricket data, failing to interpret novel attributes like "Wickets" or "Maidens."

Standard models tend to overfit to specific training schemas, limiting their ability to generalize to the vast diversity of information available on the web. This paper addresses the challenge of generating fluent text from tables containing attributes never seen during the training phase.

Methods: The AlignNet Framework

The proposed AlignNet architecture enhances a standard encoder-decoder model with a specialized Schema Aligner module.

The AlignNet architecture, featuring a Schema Aligner that maps unseen input attributes to known support set schemas before encoding.

1. Table-to-Sequence Base

The core model utilizes a standard encoder to transform attribute-value pairs into vector representations. An LSTM decoder, equipped with an attention mechanism and a copy mechanism, generates the final text. The copy mechanism is crucial as it allows the model to point directly to specific values within the table.

2. The Schema Aligner and Hungarian Algorithm

When the model encounters an input table, it randomly samples a "support set" of schemas from the training data. To bridge the gap between unseen and seen attributes, the model calculates cosine similarities between their embeddings. It then employs the Hungarian Algorithm to find an optimal bipartite matching that maximizes the total similarity score across the entire table.

3. Representation Replacement

Once an optimal match is found (e.g., aligning the unseen "Winner" to the seen "Player"), the representation of the unseen attribute is replaced by its best-aligned "seen" counterpart. This allows the decoder to leverage its pre-existing linguistic knowledge of how to describe that specific type of information. The training objective also includes a modified loss function that factors in the alignment score, encouraging the model to rely on alignments that are truly helpful for generation.

Experimental Setup

The researchers utilized the WIKITABLETEXT dataset, an open-domain corpus derived from Wikipedia. To rigorously test generalization, the authors created a new benchmark by subsampling the training set to sizes of 50, 100, 200, and 500 instances. Crucially, they ensured that over 80% of the attributes in the development and test sets remained "unseen" by the model during training.

Results and Analysis

AlignNet consistently outperformed baselines, including standard Transformer-based models and Meta-Learning (MAML) approaches, across all training set sizes.

Performance comparison across different training set sizes (50, 100, 200, 500) showing AlignNet consistently outperforming the base model, particularly in the "sweet spot" of unseen attribute ratios.

Key Insights

  • Attributes as Paraphrases: The model's success stems from the intuition that different tables share underlying concepts. "Season" in sports is functionally similar to "Year" in history. By treating these as paraphrases, AlignNet bridges the domain gap.
  • The "Sweet Spot": As shown in the performance charts, the most dramatic improvements occur when the proportion of unseen attributes is between 40% and 70%. If a table is entirely familiar or entirely alien, the relative benefit of alignment is less pronounced.
  • Quantitative Gains: On the smallest training set (50 instances), AlignNet achieved a BLEU-4 score of 18.8, compared to the base model's 14.1—a 33.3% relative improvement.
  • Qualitative Improvements: While standard models often generated "out-of-vocabulary" tokens (<unk>) for unseen attributes, AlignNet successfully verbalized relevant content into fluent sentences by mapping unknown attributes to known linguistic patterns.

Conclusion and Future Work

AlignNet demonstrates that explicit schema alignment is a powerful tool for open-world NLG. While the current approach relies primarily on the semantic similarity of attribute names, future work could explore incorporating structural information (such as table co-occurrence patterns) or handling more complex, hierarchical data structures. Refining how the model handles truly unique attributes that have no reasonable match in the support set remains an open challenge.

Summary updated

Abstract

Traditional table-to-text natural language generation (NLG) tasks focus on generating text from schemas that are already seen in the training set. This limitation curbs their generalizabilities towards real-world scenarios, where the schemas of input tables are potentially infinite. In this paper, we propose the new task of table-to-text NLG with unseen schemas, which specifically aims to test the generalization of NLG for input tables with attribute types that never appear during training. To do this, we construct a new benchmark dataset for this task. To deal with the problem of unseen attribute types, we propose a new model that first aligns unseen table schemas to seen ones, and then generates text with updated table representations. Experimental evaluation on the new benchmark demonstrates that our model outperforms baseline methods by a large margin. In addition, comparison with standard data-to-text settings shows the challenges and uniqueness of our proposed task.
BibTeX
@misc{liu2019table,
  author = {Tianyu Liu and Wei Wei and William Yang Wang},
  title = {{Table-to-Text Natural Language Generation with Unseen Schemas}},
  year = {2019},
  eprint = {1911.03601},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.03601},
}
Figure