Program Enhanced Fact Verification with Verbalization and Graph Attention Network

Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, Xiaodan Zhu

Abstract: Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, Xiaodan Zhu. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Fact verification—the task of determining whether a statement is supported or refuted by evidence—has seen significant progress in the domain of unstructured text. However, verifying statements against structured tables presents a unique set of challenges. Unlike plain text, tables require a combination of linguistic understanding and symbolic reasoning. For instance, a statement like "Ji-young Oh played more tournaments in 2008 than any other year" requires identifying the correct row, comparing values across the entire column, and understanding the concept of a maximum value.

This paper addresses two primary hurdles in table-based fact-checking:

  1. Symbolic Reasoning: Standard language models like BERT are not inherently designed to perform mathematical operations such as max, count, or average.
  2. The Semantic Gap: There is a fundamental disconnect between rigid logical programs (the symbolic representation of a query) and the fluid natural language of the statement.

To bridge this gap, the authors introduce ProgVGAT (Program-enhanced Verbalization and Graph ATtention Network), a framework that translates symbolic logic into natural language "verbalizations" and fuses them with table data using a graph-based architecture.

The ProgVGAT Framework

The ProgVGAT framework operates in three distinct stages to ensure robust reasoning and accurate verification.

1. Program Selection with Margin Loss

Because a single statement can be mapped to multiple logical programs, the model must select the most faithful representation. The authors employ a BERT-based selector trained with a novel margin loss. This strategy helps the model distinguish between truly correct programs and "spurious" ones—programs that might accidentally yield the correct True/False label through flawed logic.

2. Verbalization with Program Execution

This stage is the core innovation of the paper. Rather than feeding raw symbolic code to the model, ProgVGAT executes the program step-by-step. Each intermediate result is converted into a natural language sentence using pre-defined templates. For example, the operation eq(max(played), 29) is verbalized as "The maximum value of the column 'played' is 29." This allows the model to treat logical evidence as standard text, leveraging the pre-trained linguistic power of BERT.

3. Graph-based Verification

To synthesize the evidence, the model constructs a heterogeneous graph. This graph includes nodes for the verbalized program steps (Prog-Exec Nodes), specific table cells (Entity Nodes), and a global representation of the table and statement (Table-BERT Node). A Graph Attention Network (GAT) facilitates information exchange between these nodes, followed by a gated attention mechanism to produce the final classification.

The ProgVGAT architecture: from program selection and verbalization to graph-based verification.

Experimental Results

The model was evaluated on the TABFACT dataset, a large-scale benchmark containing over 118,000 statement-table pairs. ProgVGAT achieved a new state-of-the-art accuracy of 74.4% on the test set, outperforming previous models like LogicalFactChecker (71.7%).

Performance comparison of ProgVGAT against various baselines on the TABFACT dataset.

Key findings from the results include:

  • Complex Reasoning: On the "complex" subset of TABFACT, which requires multi-step logic, ProgVGAT reached 67.6% accuracy, a significant improvement over the 58.2% achieved by the Table-BERT baseline.
  • Ablation Insights: Removing the Graph Attention component resulted in a performance drop, confirming that the structural fusion of symbolic and linguistic evidence is vital.
  • Human-AI Gap: Despite these gains, a substantial gap remains between the model (74.4%) and human performance (92.1%), particularly on complex reasoning tasks.

Conclusion

ProgVGAT demonstrates that the key to mastering table-based fact verification lies in bridging the gap between symbolic logic and natural language. By "verbalizing" logical steps and employing graph-based attention, the model can reason more transparently and accurately. While challenges remain regarding template dependency and scalability to massive tables, ProgVGAT represents a significant step toward more robust and interpretable automated fact-checking.

Summary updated

BibTeX
@inproceedings{yang2020program,
  author = {Xiaoyu Yang and Feng Nie and Yufei Feng and Quan Liu and Zhigang Chen and Xiaodan Zhu},
  title = {{Program Enhanced Fact Verification with Verbalization and Graph Attention Network}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=vW7zqoz0vCX},
}
Figure