Introduction
Verifying factual statements against structured data, such as tables, presents a unique challenge for natural language processing. Unlike standard text-based verification, table reasoning requires "heterogeneous reasoning"—the ability to simultaneously handle textual matching, numerical comparisons, and complex logical deductions (e.g., superlatives or counting). Traditional approaches often rely on brittle program-based methods or computationally expensive table-specific pre-training.
Researchers from Tsinghua University have introduced SaMoE (Self-adaptive Mixture-of-Experts), a modular framework designed to address these challenges by delegating reasoning tasks to a committee of specialized neural experts. By building upon standard language models like RoBERTa, SaMoE achieves state-of-the-art performance without the need for massive synthetic data pre-training.
The SaMoE Framework
The SaMoE architecture is divided into three primary stages: feature extraction, expert reasoning, and adaptive management.
- Feature Extractor: To handle the context window limitations of transformers, the model prunes irrelevant columns and serializes the remaining table data into a natural language string.
- Specialized Experts: The model employs multiple identical expert networks. Through a targeted training process, these experts are encouraged to specialize in different logic types, such as numerical ranking or linguistic entailment.
- Management Module: This "brain" consists of a Manager, which assigns initial attention scores based on statement "trigger words," and a Supervisor, which adaptively adjusts these weights based on the experts' historical accuracy during training.
Expert Specialization and the "Anti-Monopoly" Loss
A significant hurdle in Mixture-of-Experts (MoE) models is the "monopoly" problem, where a single expert becomes a generalist while others remain untrained. SaMoE solves this using a Manager Assumption Loss ($L_M$). By utilizing a small pool of 26 trigger words (e.g., "most," "total," "more than"), the model forces training samples to be distributed across the expert pool.
As shown in the training dynamics, the inclusion of $L_M$ ensures that all experts converge to high accuracy, rather than relying on a single dominant network. This specialization allows the model to avoid the confusion inherent in trying to learn contradictory reasoning types within a single set of weights.
Experimental Results
The framework was evaluated on the TabFact dataset, which contains over 117,000 statements across 16,000 Wikipedia tables. SaMoE set a new benchmark for the field:
- Overall Accuracy: Achieved 85.1%, outperforming previous program-enhanced methods by 2.4%.
- Complex Reasoning: On the difficult "Complex" subset of TabFact, SaMoE reached 80.9%, a significant improvement over the previous state-of-the-art of 77.4%.
- Efficiency: By using RoBERTa-Large as a backbone, SaMoE outperformed models like TAPEX, which require extensive pre-training on synthetic SQL-to-table data.
Conclusion and Future Work
SaMoE demonstrates that a committee of specialized experts, coordinated by a self-adaptive supervisor, is highly effective for complex tabular reasoning. While the current model relies on a small manual list of trigger words to initiate specialization, future iterations could explore automated discovery of these categories. Furthermore, integrating this MoE approach into larger generative models could significantly improve the reliability of LLMs when interacting with structured data.
Summary updated