TL;DR
DataPraxis is a novel framework for synthesizing complex data analysis tasks by "evolving" workspaces—using initial findings to uncover deeper, non-obvious questions. By training on the resulting DataPraxis-5K dataset, mid-sized models (27B-35B) can match or outperform frontier LLMs on professional analytics benchmarks.
What Problem Does This Paper Address?
While large language models (LLMs) are becoming proficient at basic coding and tool use, performing professional-grade data analysis remains a high bar. Professional analysis isn't just about answering a single prompt; it requires sustained reasoning, iterative investigation, and the ability to connect dots across different datasets.
The authors identify a "data bottleneck": generating high-quality training tasks directly from raw data often results in shallow questions. Deeper insights—the kind that drive business decisions—usually only emerge after an initial layer of analysis has been performed. Furthermore, verifying the correctness of long-horizon agent trajectories is notoriously difficult. DataPraxis tackles both the task complexity and verification challenges.
Methods
The core innovation is Verified Workspace Evolution. Instead of asking an agent to generate a task from a raw CSV, the framework follows a multi-stage process:
- Workspace Evolution: A "Director" agent proposes analytical directions based on a data profile. An "Executor" performs the analysis, and a "Validator" verifies the results. This process enriches the workspace with new scripts, intermediate data, and validated findings.
- Evolution Modes: The framework uses four distinct modes to grow the "tree" of analysis: Deepen (extend findings), Search (retrieve external info), Decompose (split tasks), and Compose (merge findings from different branches).
- Backward Synthesis: Once a "deep" finding is reached (e.g., at Level 3 of the evolution), the framework works backward to the original raw data. It constructs a standalone task instruction ($q$) that requires the agent to reach that final deep conclusion starting from scratch, without seeing the intermediate steps generated during evolution.
- Dual-Evidence Verification: To ensure the training data is high-quality, the authors combine "evolution evidence" (the path taken to build the task) with "solution evidence" (trajectories from independent models) to generate robust evaluation rubrics.
Datasets
- DataPraxis-5K: A new training dataset featuring 5,000 verified trajectories across 15 domains (business, finance, sports, etc.).
- DABStep: Used for evaluating multi-step data analysis reasoning.
- JobBench: Used to assess the ability to produce professional deliverables across various occupations.
- RideBench: A newly introduced internal benchmark simulating complex business analysis in ride-hailing scenarios.
Key Insights and Ideas
-
Analysis Informs Questions: The authors found that the "perplexity" (a measure of how hard it is to predict a question) of deep questions is much lower when the model has access to prior analysis. In other words, you can't easily ask the "right" deep questions until you've done the preliminary work. Question perplexity drops significantly when models have access to prior analytical findings.
-
The Value of Depth: As the "evolution depth" increases, tasks become objectively harder. Pass rates drop and the number of required tool calls increases, providing a much richer training signal for agents. As evolution depth increases, pass rates fall and tool-call turns rise, indicating higher task difficulty.
-
Composition is Key: By merging different analytical branches (the Compose mode), the framework can create tasks that require synthesizing information from disparate sources—a hallmark of professional-level work.
Key Results
The models trained on this data, DataPraxis-27B and DataPraxis-35B-A3B, showed dramatic improvements over their base models:
- Performance Leap: DataPraxis-27B improved its average benchmark score from 48.36 to 60.39.
- Matching the Giants: On the DABStep "Hard" set, DataPraxis-27B achieved a score of 49.74, matching or surpassing frontier models like GLM-5.2 and DeepSeek-V4-Pro.
- Behavioral Shifts: Post-training, agents became significantly better at self-verification (inspecting their own outputs) and explicit task organization (breaking complex requests into subgoals).
Limitations and Future Work
- Seed Dependency: The framework relies on "seed" workspaces that are complex enough to support deep exploration. If the initial data is too simple, the evolution "tree" won't grow very deep.
- Teacher Model Quality: The difficulty of synthesized tasks and the quality of training trajectories are still bounded by the capabilities of the "teacher" models used during the evolution and synthesis phases.
- Future Directions: The authors suggest that scaling this approach to even more diverse environments and higher-fidelity simulations (like RideBench) could further push the boundaries of agentic data analysis.
Summary updated