Program Enhanced Fact Verification with Verbalization and Graph Attention Network

Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, Xiaodan Zhu

Abstract: Performing fact verification based on structured data is important for many real-life applications and is a challenging research problem, particularly when it involves both symbolic operations and informal inference based on language understanding. In this paper, we present a Program-enhanced Verbalization and Graph Attention Network (ProgVGAT) to integrate programs and execution into textual inference models. Specifically, a verbalization with program execution model is proposed to accumulate evidences that are embedded in operations over the tables. Built on that, we construct the graph attention verification networks, which are designed to fuse different sources of evidences from verbalized program execution, program structures, and the original statements and tables, to make the final verification decision. To support the above framework, we propose a program selection module optimized with a new training strategy based on margin loss, to produce more accurate programs, which is shown to be effective in enhancing the final verification results. Experimental results show that the proposed framework achieves the new state-of-the-art performance, a 74.4% accuracy, on the benchmark dataset TABFACT.

Introduction

Verifying claims against structured tables requires a sophisticated combination of natural language understanding and symbolic reasoning. For example, a statement like "Ji-young Oh played more tournaments in 2008 than any other year" requires filtering rows, finding a maximum value, and performing a comparison.

Existing approaches often struggle with this "semantic gap." Neural models like Table-BERT, which flatten tables into text, frequently fail at complex symbolic operations. Conversely, purely symbolic systems are often too rigid to handle the nuances of natural language. This paper introduces ProgVGAT, a framework designed to bridge this gap by translating logical programs into natural language "evidence sentences" and using a Graph Attention Network to fuse these insights with the original table data.

Methodology

The ProgVGAT framework consists of three primary stages: Program Selection, Verbalization with Program Execution, and Graph-based Verification.

Overview of the ProgVGAT framework, showing program selection, verbalization of execution steps, and the graph attention network for final verification.

Program Selection with Margin Loss

The system first generates candidate logical programs representing the statement. To identify the most semantically correct program, the authors employ a novel training strategy based on margin loss. This approach helps the model distinguish between programs that yield the correct answer for the right reasons versus those that are merely "lucky" (spurious programs).

Verbalization with Program Execution

A key innovation of this work is the verbalization of symbolic execution. Instead of feeding raw code to a neural network, the model executes the program step-by-step and converts each operation into a natural language sentence using pre-defined templates. For instance, a max operation is verbalized as: "The max value of column tournaments played is 29." This transforms abstract logic into a format that pre-trained models like BERT can effectively process.

Graph Attention Verification Network

To reach a final decision (Entailed vs. Refuted), the model constructs a heterogeneous graph with three node types: Prog-Exec Nodes (verbalized execution steps), Entity Nodes (specific table cells or values), and a Table-BERT Node (global representation of the original table and statement). A Graph Attention Network (GAT) propagates information across these nodes, allowing the model to reason over the program's hierarchy and the table's content simultaneously.

Experimental Results

The model was evaluated on TABFACT, a large-scale dataset containing over 117,000 manually annotated statements based on Wikipedia tables. The dataset includes both "simple" and "complex" subsets.

Performance comparison of ProgVGAT against various baselines on the TABFACT dataset.

ProgVGAT achieved a new state-of-the-art 74.4% accuracy on the TABFACT test set, significantly outperforming the previous best model, LogicalFactChecker (71.7%). Notably, on the "complex" subset, ProgVGAT reached 67.6% accuracy, a nearly 17-point improvement over the Table-BERT baseline. Ablation studies confirmed that the graph attention component is vital, as its removal led to a 1% drop in accuracy.

Discussion

The success of ProgVGAT highlights several key insights. First, verbalization acts as a bridge, allowing the model to leverage the linguistic knowledge of pre-trained models like BERT. Second, structural awareness provided by the Graph Attention Network enables the model to understand the hierarchy of logical programs. Finally, the use of margin loss effectively targets the "spurious program" problem, ensuring the model selects programs that truly match the statement's meaning.

Conclusion

While ProgVGAT sets a new record for AI performance on TABFACT, a significant gap remains compared to human performance (92.1%). Current limitations include a reliance on pre-defined templates for verbalization and challenges with extremely complex, multi-step manipulations. Future research may explore learned natural language generation for verbalization and more sophisticated methods for exploring the program search space.

Summary updated

Abstract

Performing fact verification based on structured data is important for many real-life applications and is a challenging research problem, particularly when it involves both symbolic operations and informal inference based on language understanding. In this paper, we present a Program-enhanced Verbalization and Graph Attention Network (ProgVGAT) to integrate programs and execution into textual inference models. Specifically, a verbalization with program execution model is proposed to accumulate evidences that are embedded in operations over the tables. Built on that, we construct the graph attention verification networks, which are designed to fuse different sources of evidences from verbalized program execution, program structures, and the original statements and tables, to make the final verification decision. To support the above framework, we propose a program selection module optimized with a new training strategy based on margin loss, to produce more accurate programs, which is shown to be effective in enhancing the final verification results. Experimental results show that the proposed framework achieves the new state-of-the-art performance, a 74.4% accuracy, on the benchmark dataset TABFACT.
BibTeX
@inproceedings{yang2020program,
  author = {Xiaoyu Yang and Feng Nie and Yufei Feng and Quan Liu and Zhigang Chen and Xiaodan Zhu},
  title = {{Program Enhanced Fact Verification with Verbalization and Graph Attention Network}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=vW7zqoz0vCX},
}
Figure