Bridge the Gap between Language models and Tabular Understanding

Nuo Chen, Linjun Shou, Ming Gong, Jian Pei, Chenyu You, Jianhui Chang, Daxin Jiang, Jia LiarXiv 2023 · OpenReview

Abstract

Table pretrain-then-finetune paradigm has been proposed and employed at a rapid pace after the success of pre-training in the natural language domain. Despite the promising findings in tabular pre-trained language models (TPLMs), there is an input gap between pre-training and fine-tuning phases. For instance, TPLMs jointly pre-trained with table and text input could be effective for tasks also with table-text joint input like table question answering, but it may fail for tasks with only tables or text as input such as table retrieval. To this end, we propose UTP, an approach that dynamically supports three types of multi-modal inputs: table-text, table, and text. Specifically, UTP is pre-trained with two strategies: (1) We first utilize a universal mask language modeling objective on each kind of input, enforcing the model to adapt various inputs. (2) We then present Cross-Modal Contrastive Regularization (CMCR), which utilizes contrastive learning to encourage the consistency between table-text cross-modality representations via unsupervised instance-wise training signals during pre-training. By these means, the resulting model not only bridges the input gap between pre-training and fine-tuning but also advances in the alignment of table and text. Extensive results show UTP achieves superior results on uni-modal input tasks (e.g., table retrieval) and cross-modal input tasks (e.g., table question answering).

1. Introduction

The success of pre-training in natural language processing has led to the rapid development of Tabular Pre-trained Language Models (TPLMs). However, a significant "input gap" persists between pre-training and fine-tuning phases. Most TPLMs, such as TAPAS or TaBERT, are pre-trained on joint table-text inputs. While effective for tasks like table question answering (QA), these models often struggle with uni-modal tasks like table retrieval, where only text or tables are provided as input. Furthermore, standard objectives like Masked Language Modeling (MLM) focus on local token relationships, often neglecting the global semantic alignment between tables and their associated text.

2. Methodology

We propose Universal Table-text Pre-training (UTP) to address these limitations through two key strategies: Universal MLM and Cross-Modal Contrastive Regularization (CMCR).

2.1 Universal Masked Language Modeling

UTP employs a shared encoder to process three distinct input modalities: text-only ($X_W$), table-only ($X_T$), and joint table-text ($X_{WT}$). By applying the MLM objective across all three formats, the model learns to be input-agnostic, facilitating better performance on diverse downstream tasks regardless of the input format.

2.2 Cross-Modal Contrastive Regularization (CMCR)

To capture global semantic connections, we introduce CMCR. This strategy uses contrastive learning to align the representations of text ($r_w$), tables ($r_t$), and joint pairs ($r_{wt}$). By encouraging these representations to be similar for the same data instance while distancing them from others in the batch, UTP ensures that the model understands the high-level semantic equivalence between a table and its description.

The UTP architecture featuring Universal MLM and CMCR strategies.

3. Experiments and Results

UTP was pre-trained on a corpus of 1.5 million parallel table-text pairs from English Wikipedia. We evaluated the model on two primary tasks: Table Retrieval and Table Question Answering.

3.1 Table Retrieval

On the NQ-TABLES benchmark, UTP significantly outperformed existing baselines. As shown in the results table, UTP achieved a Recall@1 of 38.45, a substantial improvement over TAPAS (28.95). With the addition of hard negative (hn) mining, UTP reached a state-of-the-art Recall@1 of 50.54.

Comparison of UTP against baselines on the NQ-TABLES retrieval task.

3.2 Table Question Answering

Evaluation on the WIKISQL dataset demonstrated UTP's versatility. In a weakly supervised setting, UTP achieved 84.5% accuracy, and 88.1% with full supervision, consistently surpassing the performance of the TAPAS model. Ablation studies confirmed that the CMCR component is vital for these gains, particularly in retrieval tasks where global alignment is paramount.

4. Conclusion

UTP bridges the input gap in tabular understanding by supporting multi-modal inputs and enforcing cross-modal alignment. The model achieves state-of-the-art results on both uni-modal and cross-modal tasks. Future research will explore scaling the pre-training data and extending UTP to multilingual contexts.

Summary updated

Abstract

Table pretrain-then-finetune paradigm has been proposed and employed at a rapid pace after the success of pre-training in the natural language domain. Despite the promising findings in tabular pre-trained language models (TPLMs), there is an input gap between pre-training and fine-tuning phases. For instance, TPLMs jointly pre-trained with table and text input could be effective for tasks also with table-text joint input like table question answering, but it may fail for tasks with only tables or text as input such as table retrieval. To this end, we propose UTP, an approach that dynamically supports three types of multi-modal inputs: table-text, table, and text. Specifically, UTP is pre-trained with two strategies: (1) We first utilize a universal mask language modeling objective on each kind of input, enforcing the model to adapt various inputs. (2) We then present Cross-Modal Contrastive Regularization (CMCR), which utilizes contrastive learning to encourage the consistency between table-text cross-modality representations via unsupervised instance-wise training signals during pre-training. By these means, the resulting model not only bridges the input gap between pre-training and fine-tuning but also advances in the alignment of table and text. Extensive results show UTP achieves superior results on uni-modal input tasks (e.g., table retrieval) and cross-modal input tasks (e.g., table question answering).
BibTeX
@misc{chen2023bridge,
  author = {Nuo Chen and Linjun Shou and Ming Gong and Jian Pei and Chenyu You and Jianhui Chang and Daxin Jiang and Jia Li},
  title = {{Bridge the Gap between Language models and Tabular Understanding}},
  year = {2023},
  eprint = {2302.09302},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.09302},
}
Figure