Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities in general-purpose programming, yet their proficiency in specialized domains like statistical analysis remains under-explored. This study presents a rigorous human-expert evaluation of GPT-3.5, GPT-4.0, and Llama 3.1 70B in the context of SAS programming. By evaluating 207 statistical tasks across 10 criteria, we identify a significant "syntax-logic gap": models produce readable code but frequently fail in execution and output accuracy.
Introduction
While LLMs have revolutionized general software engineering, their reliability in statistical programming remains a black box. Statistical analysis requires more than just valid syntax; it demands deep domain knowledge to select appropriate models and interpret results accurately. This paper establishes a human-gold-standard framework to evaluate if AI can be trusted with data science tasks, specifically focusing on SAS—a language widely used in highly regulated industries like finance and pharmaceuticals.
Methodology
The researchers developed a multi-dimensional evaluation framework to move beyond automated metrics like BLEU, which often fail to capture functional correctness in code.
The framework involved a panel of over 10 statistics faculty and PhD candidates who conducted blinded evaluations of AI-generated code. Each script was rated on 10 criteria across three categories: Code Quality (structure and readability), Executability (running without errors), and Output Quality (accuracy of results). Regression models with non-parametric bootstrapping and random effects models were employed to ensure statistical significance and account for potential rater bias.
Experimental Setup and Datasets
The study utilized a newly curated benchmark consisting of 65 datasets from diverse fields (biology, engineering, social sciences) and 207 statistical tasks ranging from undergraduate to master's level difficulty. Each task included a human-verified "gold standard" SAS script for comparison.
Key Insights and Results
The evaluation revealed several critical findings regarding the current state of LLMs in statistics:
- The "B" Student Phenomenon: If these LLMs were students in a statistics course, they would all earn a "B." They are competent but not yet reliable enough for autonomous work.
- The Syntax-Logic Gap: There is a massive drop-off in performance across categories. While models scored ~94% on "looking" correct (Group 1), they only achieved ~61% on actually running (Group 2) and ~52% on producing the correct statistical output (Group 3).
- Model Comparison: While GPT-4.0 achieved the highest mean score (37.75/50), the difference between it, Llama (37.26), and GPT-3.5 (37.15) was not statistically significant. Common failure points included hallucinating variable names and missing SAS-specific operators.
Conclusion
Current LLMs are useful assistants but require expert oversight in statistical programming. Future research will expand this framework to R programming and explore automated metrics that better reflect the nuances of statistical logic.
Summary updated