Table-based Fact Verification with Self-adaptive Mixture of Experts

Yuxuan Zhou, Xien Liu, Kaiyin Zhou, Ji Wu

Introduction

Verifying factual statements against structured data, such as tables, presents a unique challenge for natural language processing. Unlike standard text-based verification, table reasoning requires "heterogeneous reasoning"—the ability to simultaneously handle textual matching, numerical comparisons, and complex logical deductions (e.g., superlatives or counting). Traditional approaches often rely on brittle program-based methods or computationally expensive table-specific pre-training.

Researchers from Tsinghua University have introduced SaMoE (Self-adaptive Mixture-of-Experts), a modular framework designed to address these challenges by delegating reasoning tasks to a committee of specialized neural experts. By building upon standard language models like RoBERTa, SaMoE achieves state-of-the-art performance without the need for massive synthetic data pre-training.

The SaMoE Framework

The SaMoE architecture is divided into three primary stages: feature extraction, expert reasoning, and adaptive management.

The SaMoE framework architecture, featuring the Feature Extractor and the Multi-expert Reasoning module.
  1. Feature Extractor: To handle the context window limitations of transformers, the model prunes irrelevant columns and serializes the remaining table data into a natural language string.
  2. Specialized Experts: The model employs multiple identical expert networks. Through a targeted training process, these experts are encouraged to specialize in different logic types, such as numerical ranking or linguistic entailment.
  3. Management Module: This "brain" consists of a Manager, which assigns initial attention scores based on statement "trigger words," and a Supervisor, which adaptively adjusts these weights based on the experts' historical accuracy during training.

Expert Specialization and the "Anti-Monopoly" Loss

A significant hurdle in Mixture-of-Experts (MoE) models is the "monopoly" problem, where a single expert becomes a generalist while others remain untrained. SaMoE solves this using a Manager Assumption Loss ($L_M$). By utilizing a small pool of 26 trigger words (e.g., "most," "total," "more than"), the model forces training samples to be distributed across the expert pool.

Expert accuracy over training steps when using the combined loss function, demonstrating successful specialization across all five experts.

As shown in the training dynamics, the inclusion of $L_M$ ensures that all experts converge to high accuracy, rather than relying on a single dominant network. This specialization allows the model to avoid the confusion inherent in trying to learn contradictory reasoning types within a single set of weights.

Experimental Results

The framework was evaluated on the TabFact dataset, which contains over 117,000 statements across 16,000 Wikipedia tables. SaMoE set a new benchmark for the field:

  • Overall Accuracy: Achieved 85.1%, outperforming previous program-enhanced methods by 2.4%.
  • Complex Reasoning: On the difficult "Complex" subset of TabFact, SaMoE reached 80.9%, a significant improvement over the previous state-of-the-art of 77.4%.
  • Efficiency: By using RoBERTa-Large as a backbone, SaMoE outperformed models like TAPEX, which require extensive pre-training on synthetic SQL-to-table data.

Conclusion and Future Work

SaMoE demonstrates that a committee of specialized experts, coordinated by a self-adaptive supervisor, is highly effective for complex tabular reasoning. While the current model relies on a small manual list of trigger words to initiate specialization, future iterations could explore automated discovery of these categories. Furthermore, integrating this MoE approach into larger generative models could significantly improve the reliability of LLMs when interacting with structured data.

Summary updated

Abstract

The table-based fact verification task has recently gained widespread attention and yet remains to be a very challenging problem. It inherently requires informative reasoning over natural language together with different numerical and logical reasoning on tables (e.g., count, superlative, comparative). Considering that, we exploit mixture-of-experts and present in this paper a new method: Self-adaptive Mixture-of-Experts Network (SaMoE). Specifically, we have developed a mixture-of-experts neural network to recognize and execute different types of reasoning -- the network is composed of multiple experts, each handling a specific part of the semantics for reasoning, whereas a management module is applied to decide the contribution of each expert network to the verification result. A self-adaptive method is developed to teach the management module combining results of different experts more efficiently without external knowledge. The experimental results illustrate that our framework achieves 85.1% accuracy on the benchmark dataset TabFact, comparable with the previous state-of-the-art models. We hope our framework can serve as a new baseline for table-based verification. Our code is available at https://github.com/THUMLP/SaMoE.
BibTeX
@inproceedings{zhou2022table,
  author = {Yuxuan Zhou and Xien Liu and Kaiyin Zhou and Ji Wu},
  title = {{Table-based Fact Verification with Self-adaptive Mixture of Experts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.13},
}
Figure