Abstract
Multimodal Large Language Models (MLLMs) often struggle with "WildStruct" tables—real-world data characterized by visual degradation, complex hierarchies, and symbolic clutter. We present TableMoE, a novel architecture that employs neuro-symbolic routing to decompose table understanding into specialized sub-tasks. By predicting the semantic roles of visual tokens and routing them to expert connectors (HTML, JSON, and Code), TableMoE achieves state-of-the-art performance on challenging benchmarks, outperforming proprietary models like GPT-4o while offering an interpretable reasoning trace.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in processing unstructured text, their ability to interpret structured data, particularly tables, remains a significant challenge. Real-world tables are rarely clean grids; they frequently feature "WildStruct" complications such as motion blur, perspective skew, nested cell hierarchies, and dense symbolic content like mathematical formulas.
Current state-of-the-art Multimodal LLMs (MLLMs) typically adopt a monolithic approach, treating table images as a single sequence of visual tokens. This entanglement of layout, semantics, and symbolic structure often leads to brittleness and hallucinations, especially when faced with visual noise. These models lack a dedicated mechanism to parse the underlying structural logic of a table before attempting to reason over its content.
Methodology: Neuro-Symbolic Routing
TableMoE addresses these limitations by replacing the standard linear connector between the vision encoder and the LLM with a Mixture-of-Connector-Experts (MoCE). This architecture is governed by a Neuro-Symbolic Routing mechanism designed to provide a "semantic planner" for the model.
The process involves four key stages:
- Latent Semantic Role Prediction: A lightweight classifier identifies the semantic role of each visual token (e.g.,
HEADER,DATA,AXIS, orFORMULA). - Specialized Expert Routing: Tokens are routed to one of three experts pre-trained on specific alignment tasks:
- Table-to-HTML: Optimized for structural layout and hierarchical relationships.
- Table-to-JSON: Focused on spatial triples and fine-grained visual attributes.
- Table-to-Code: Translates table data into executable Python/Matplotlib code for symbolic abstraction and visualization.
- Confidence-Aware Gating: To prevent hallucinations in low-quality images, the model uses Shannon entropy to measure uncertainty. If role prediction is ambiguous, the system defaults to a "generalist" expert.
- Curriculum Annealing: A specialized fine-tuning protocol progressively introduces "WildStruct" noise, hardening the routing mechanism against real-world degradations.
Data and Benchmarking
To support this architecture, the authors developed TableMoE-Align, a massive pre-training corpus containing 1.2 million table–HTML–JSON–code quadruples across diverse domains like finance and biomedicine.
Furthermore, the study introduces the WildStruct Benchmarks (WMMFinQA, WMMTatQA, WMMTabDialog, and WMMFinanceMath), specifically designed to stress-test models against visual degradation and complex reasoning tasks.
Results and Discussion
TableMoE demonstrates superior performance across both standard and "WildStruct" benchmarks. On the WildStruct-Bench, it achieved an average accuracy of 80.31%, surpassing GPT-4o by 5.2 percentage points. Its performance on MMMU-Table was even more pronounced, reaching 54.53% accuracy—a 10.55-point lead over GPT-4o.
Ablation studies confirm the importance of the neuro-symbolic planner; removing the routing graph resulted in a 5.37% performance drop. Notably, TableMoE exhibits a unique "abstention" behavior: when faced with extreme blur, it recognizes high entropy in its predictions and avoids the confident hallucinations common in other models, instead signaling a need for clearer input.
Conclusion
TableMoE represents a significant step toward robust multimodal table understanding. By decomposing the problem into semantic role prediction and specialized expert routing, the model gains both accuracy and interpretability. Future work will focus on making the semantic taxonomy more adaptive and exploring the scalability of this neuro-symbolic approach with larger foundation models.
Summary updated