Abstract
Querying tables with unstructured data is challenging due to the presence of text (or image), either embedded in the table or in external paragraphs, which traditional SQL struggles to process, especially for tasks requiring semantic reasoning. While Large Language Models (LLMs) excel at understanding context, they face limitations with long input sequences. Existing approaches that combine SQL and LLMs typically rely on rigid, predefined workflows, limiting their adaptability to complex queries. To address these issues, we introduce Weaver, a modular pipeline that dynamically integrates SQL and LLMs for table-based question answering (TableQA). Weaver generates a flexible, step-by-step plan that combines SQL for structured data retrieval with LLMs for semantic processing. By decomposing complex queries into manageable subtasks, Weaver improves accuracy and generalization. Our experiments show that Weaver consistently outperforms state-of-the-art methods across four TableQA datasets, reducing both API calls and error rates.
Introduction
In the modern data landscape, tables are rarely composed of purely structured data. Real-world datasets often contain a "hybrid" mix of structured fields (like prices, dates, or IDs) and unstructured content (like free-form text descriptions or associated images). Traditional tools like SQL are highly efficient at structured logic but remain "blind" to semantic nuances. Conversely, Large Language Models (LLMs) possess deep semantic understanding but frequently struggle with complex, multi-step numerical reasoning or large-scale data aggregation—a phenomenon often described as the "lost in the middle" effect.
Weaver addresses this "reasoning gap" by interweaving the strengths of both paradigms. It is a modular framework designed to master complex table reasoning by dynamically alternating between SQL for data retrieval and LLMs for semantic inference.
The Weaver Framework
Weaver operates through a structured, four-phase pipeline designed to ensure both accuracy and efficiency:
- Pre-processing: The system first cleans the table schema to prevent SQL reserved-word conflicts (e.g., renaming a column titled "Rank"). It then utilizes an LLM to identify and prune columns irrelevant to the specific query, reducing the noise passed to subsequent steps.
- Planning: An LLM generates a multi-step, plain-text execution plan. Each sub-task is categorized as either a SQL Step (for filtering, sorting, or aggregating) or an LLM Step (for semantic inference, such as "determine the country of origin for this specific driver").
- Plan Verification: To ensure reliability, a secondary LLM acts as a "critic." This agent checks the generated plan for logical consistency, formatting errors, and feasibility before any code is executed.
- Code Execution: The verified plan is translated into executable MySQL queries and targeted LLM prompts.
Strategic Optimizations
To overcome the limitations of standard hybrid systems, Weaver employs two key technical optimizations:
- SQL Reordering: Weaver prioritizes SQL filters at the start of the execution chain. By using SQL to prune the table size before passing data to the LLM, the system significantly reduces token consumption and minimizes the risk of LLM hallucinations caused by over-extended contexts.
- Chunk-based Parallelism: Rather than processing rows individually (which is slow) or all at once (which risks exceeding context limits), Weaver processes data in optimized, parallelized "chunks," balancing speed with context management.
Experimental Results
Weaver was evaluated against prominent benchmarks, including WikiTQ, TabFact, and FinQA, as well as multimodal sets (FinQA_MM, OTT-QA_MM) that pair tables with external text or images. The researchers specifically focused on "hybrid" subsets that necessitate both SQL logic and semantic reasoning.
Performance and Accuracy
Weaver consistently outperformed state-of-the-art baselines such as Binder, BlendSQL, and ProTrix across various backbone models (GPT-4o, Gemini 2.0 Flash, and DeepSeek-R1):
- WikiTQ_hybrid: Weaver outperformed the best baseline (ProTrix) by 5.5%.
- TabFact_hybrid: Utilizing the DeepSeek-R1 backbone, Weaver achieved a breakthrough accuracy of 91.2%.
- Robustness: In tests involving large tables (30+ rows), Weaver maintained an accuracy of 65.6%, while competing methods like ProTrix saw performance drops to as low as 37.5%.
Efficiency and Traceability
Beyond accuracy, Weaver is significantly more resource-efficient. It averages approximately 5.5 API calls per query, a stark contrast to self-consistency methods like Binder, which can require up to 50 calls. Furthermore, because Weaver generates an intermediate table at every step of the execution, the reasoning process is fully transparent and traceable, allowing for easier debugging and human verification.
Discussion and Limitations
The success of Weaver highlights the power of decoupling logical planning from code execution. By treating SQL as a sophisticated pre-processor for the LLM, the system avoids the "hybrid penalty" where performance typically drops when external knowledge is required.
However, several avenues for future work remain. The current implementation is optimized for English-language tables and single-table queries. Future iterations aim to extend the planner to handle complex relational joins across multiple tables and hierarchical (nested) table structures. Additionally, the team plans to integrate Vision-Language Models (VLMs) more deeply into the SQL planning loop to enhance multimodal reasoning capabilities.
Conclusion
Weaver represents a significant advancement in TableQA by providing a flexible, transparent, and efficient bridge between structured logic and semantic understanding. By interweaving SQL and LLMs, it provides a scalable solution for querying the messy, hybrid data structures common in real-world applications.
Summary updated