Structure-Aware Pre-Training for Table-to-Text Generation

Xinyu Xing, Xiaojun Wan

Abstract

Generating natural language descriptions from structured tables is a challenging task for standard large language models (LLMs) due to their inherent "structure-blindness." We present STTP (Structure-aware Table-to-text Pre-trained model), a framework developed by researchers at Peking University that incorporates spatial logic into the pre-training phase. By utilizing three novel self-supervised tasks—Masked Table Language Model (MTLM), Adjacent Cell Prediction (ACP), and Context Reconstruction (CR)—STTP learns to encode 2D table structures and align them with natural language. Evaluation on the WEATHERGOV and WebNLG benchmarks demonstrates that STTP significantly outperforms existing models, achieving new state-of-the-art results.

Introduction

Large language models like BART and T5 excel at generating fluent text by learning from massive linear corpora. However, they often struggle with "Table-to-Text" generation, where the input is structured data such as spreadsheets or RDF triples. In a table, a cell's meaning is derived from its spatial relationship with neighboring rows and columns. Standard models typically flatten this data into a 1D sequence, losing critical structural information and leading to factual inaccuracies or "hallucinations" in the generated text.

Methodology: Structure-Aware Pre-Training

STTP addresses this limitation by augmenting the BART architecture with a structure-aware pre-training phase. The model employs special tokens (e.g., [cell], [row], [TABLE]) to linearize tables while preserving organizational cues. The core of the approach lies in three self-supervised tasks:

  1. Masked Table Language Model (MTLM): The model predicts masked tokens (15%) within the table to learn the basic vocabulary and value distributions of tabular data.
  2. Adjacent Cell Prediction (ACP): To capture spatial logic, the model is tasked with predicting the content of cells adjacent (Up, Down, Left, Right) to a target cell. This explicitly forces the encoder to represent 2D grid relationships.
  3. Context Reconstruction (CR): This task aligns tabular data with natural language by requiring the model to reconstruct original fluent text from a table and its "noised" surrounding context.
Overview of the three structure-aware pre-training tasks: Masked Table Language Model (MTLM), Adjacent Cell Prediction (ACP), and Context Reconstruction (CR).

During the MTLM and ACP phases, the BART decoder is kept frozen to focus training on the encoder's structural representation capabilities. The decoder is subsequently unfrozen for the CR task to facilitate cross-modal alignment.

Experimental Evaluation

The model was pre-trained on a filtered subset of the WDCWebTable Corpus, consisting of 800,000 tables with high-quality context. Evaluation was conducted on two primary benchmarks:

  • WEATHERGOV: Weather records paired with natural language forecasts.
  • WebNLG: Mapping RDF triples to fluent descriptions.

Results and Discussion

STTP demonstrated superior performance across all metrics. On the WEATHERGOV dataset, it achieved a BLEU score of 82.63, surpassing the base BART model (81.54). On the WebNLG benchmark, STTP reached a BLEU score of 64.92, outperforming specialized models such as KGPT and GCN-based architectures.

Performance comparison on the WebNLG benchmark. STTP (64.92 BLEU) outperforms previous state-of-the-art models like KGPT-Seq and GCN.

Human evaluations further confirmed these findings, with STTP receiving higher scores for readability, accuracy, and overall quality. Notably, the model showed a significant reduction in data hallucinations, attributed to the spatial awareness gained through the ACP task.

Conclusion

STTP proves that explicitly modeling the 2D structure of tables is more effective than simply scaling up general-purpose language models for table-to-text tasks. Future work will focus on expanding the pre-training corpus and developing tasks to handle more complex table structures, such as hierarchical headers and multi-line cells.

Summary updated

BibTeX
@inproceedings{xing2021structure,
  author = {Xinyu Xing and Xiaojun Wan},
  title = {{Structure-Aware Pre-Training for Table-to-Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.200},
}
Figure