TL;DR
This paper introduces Multi-engine Joint State Data Analysis, a framework that bridges the gap between LLM reasoning and precise computation by embedding a persistent "data context" into the conversation. By combining LLMs with four specialized engines (SQL, Cypher, Python, and R) and a deterministic grounding method, the authors achieve over 90% accuracy on complex data tasks, outperforming current single-shot state-of-the-art models by more than 20 percentage points.
What Problem Does This Paper Address?
The authors argue that modern AI data analysis is fractured into three "worlds" that see only half the picture:
- LLMs understand semantics but cannot compute precisely (they hallucinate math).
- Business Intelligence (BI) tools aggregate precisely but are stateless and lack semantic depth.
- Machine Learning (ML) handles large-scale data on stateful kernels but lacks the business reasoning of an LLM.
Current "Text-to-SQL" or "Code Interpreter" solutions often treat analysis as a one-shot "sandbox" execution. If you ask a follow-up question, the system starts from scratch. Furthermore, autonomous agents often waste massive amounts of time and tokens "exploring" database schemas through trial and error, which is both expensive and prone to failure.
Methods
The core innovation is the Multi-engine Joint State Data Analysis architecture. It moves beyond a simple "conversational context" (what was said) to a "data context" (what is currently in memory).
Key components include:
- Stateful Data Analysis: Unlike stateless "query-and-discard" tools, this framework keeps DataFrames, variables, and models resident in memory across multiple rounds of conversation.
- The M+S Mechanism: To eliminate the cost of "exploration," the authors use a deterministic grounding method. They inject two resource files in one shot:
schema.json(the authoritative structure) andnotes.md(accumulated domain expertise and SQL conventions). - Cross-Engine State Flow: The framework allows data to flow between heterogeneous engines. For example, an analyst can pull relational data via SQL, pass it to a Python kernel for feature engineering, and then use an R kernel for specialized visualization—all within the same persistent context.
The authors structure this novelty using a four-dimensional coordinate system: The 4D Coordinate System of the Novelty
Datasets
The framework was validated across several rigorous benchmarks:
- SQL: Used the
works_cycles(AdventureWorks) database and two "dirty" databases from the BIRD benchmark (financialandformula_1) to test real-world, messy data conditions. - Graph: Evaluated on the
CypherBenchmovie subgraph (401 questions). - ML: Tested on the
TabReDweather dataset (16.95 million rows) and Shanghai housing price data (270k rows).
Key Insights and Ideas
- The "Weak-Model Amplifier": One of the most striking findings is that this mechanism provides a larger performance boost to "weaker" models. While flagship models saw a +4pp gain, smaller models like GLM-5.1 saw their accuracy jump by nearly 30 points.
- ER Diagrams are "Noise": Counterintuitively, the authors found that for strong models, adding complex Entity-Relationship (ER) diagrams actually decreased reliability by inducing "drift" (hallucinations about join options). A clean data dictionary is often superior.
- Separation of Concerns: The LLM acts strictly as the "orchestrator" (the brain), while the engines handle the "computation" (the muscle). This separation ensures the system remains robust even as LLMs evolve.
- Monotonic Convergence: The authors demonstrated that as users add "notes" to the
notes.mdfile, the system's accuracy improves predictably and stays high, turning "experience accumulation" into a reproducible engineering process.
Key Results
The results show a significant lead over existing methods:
- SQL Accuracy: Achieved 91.3% on AdventureWorks, 95.3% on financial data, and 94.6% on formula_1 data. This is roughly 20–23 points higher than the single-shot SOTA (74.45%).
- Efficiency: The M+S method halved the number of rounds required and cut token usage by nearly 40% compared to pure exploration agents.
- Graph Performance: Reached 96.5% execution accuracy on CypherBench, exceeding previous SOTA by 24–35 points.
- Scale: Successfully ran a ten-million-scale GBDT model in R by using a
float32direct channel to bypass R's memory limitations.
Limitations and Future Work
The authors are transparent about several "honesty boundaries":
- Cold Start: A brand-new database without any
notes.mdstarts at "SOTA" levels (~74%) and requires a period of use to converge to 90%+. - Qualitative Gaps: While the "joint" nature of the engines was proven, some cross-domain tasks (like linking RDB to GraphDB) are currently limited by the quality of the underlying data governance.
- Future Directions: The authors plan to extend the framework to the full CypherBench, automate the migration of "warehouse-general" rules to compress the cold-start cycle, and verify the "weak-model amplifier" effect across more international closed models like GPT-4o and Claude 3.5.
Summary updated