Performance Evaluation of Large Language Models in Statistical Programming

Xinyi Song, Kexin Xie, Lina Lee, Ruizhe Chen, Jared M. Clark, Hao He, Jie Min, Xinlei Zhang, Simin Zheng, Zhiyang Zhang, Xinwei Deng, Yili HongarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in general-purpose programming, yet their proficiency in specialized domains like statistical analysis remains under-explored. This study presents a rigorous human-expert evaluation of GPT-3.5, GPT-4.0, and Llama 3.1 70B in the context of SAS programming. By evaluating 207 statistical tasks across 10 criteria, we identify a significant "syntax-logic gap": models produce readable code but frequently fail in execution and output accuracy.

Introduction

While LLMs have revolutionized general software engineering, their reliability in statistical programming remains a black box. Statistical analysis requires more than just valid syntax; it demands deep domain knowledge to select appropriate models and interpret results accurately. This paper establishes a human-gold-standard framework to evaluate if AI can be trusted with data science tasks, specifically focusing on SAS—a language widely used in highly regulated industries like finance and pharmaceuticals.

Methodology

The researchers developed a multi-dimensional evaluation framework to move beyond automated metrics like BLEU, which often fail to capture functional correctness in code.

The evaluation setup and rating procedure for assessing LLM-generated SAS code.

The framework involved a panel of over 10 statistics faculty and PhD candidates who conducted blinded evaluations of AI-generated code. Each script was rated on 10 criteria across three categories: Code Quality (structure and readability), Executability (running without errors), and Output Quality (accuracy of results). Regression models with non-parametric bootstrapping and random effects models were employed to ensure statistical significance and account for potential rater bias.

Experimental Setup and Datasets

The study utilized a newly curated benchmark consisting of 65 datasets from diverse fields (biology, engineering, social sciences) and 207 statistical tasks ranging from undergraduate to master's level difficulty. Each task included a human-verified "gold standard" SAS script for comparison.

Examples of SAS code tasks used in the evaluation benchmark.

Key Insights and Results

The evaluation revealed several critical findings regarding the current state of LLMs in statistics:

  • The "B" Student Phenomenon: If these LLMs were students in a statistics course, they would all earn a "B." They are competent but not yet reliable enough for autonomous work.
  • The Syntax-Logic Gap: There is a massive drop-off in performance across categories. While models scored ~94% on "looking" correct (Group 1), they only achieved ~61% on actually running (Group 2) and ~52% on producing the correct statistical output (Group 3).
  • Model Comparison: While GPT-4.0 achieved the highest mean score (37.75/50), the difference between it, Llama (37.26), and GPT-3.5 (37.15) was not statistically significant. Common failure points included hallucinating variable names and missing SAS-specific operators.

Conclusion

Current LLMs are useful assistants but require expert oversight in statistical programming. Future research will expand this framework to R programming and explore automated metrics that better reflect the nuances of statistical logic.

Summary updated

BibTeX
@misc{song2025performance,
  author = {Xinyi Song and Kexin Xie and Lina Lee and Ruizhe Chen and Jared M. Clark and Hao He and Haoran He and Jie Min and Xinlei Zhang and Simin Zheng and Zhiyang Zhang and Xinwei Deng and Yili Hong},
  title = {{Performance Evaluation of Large Language Models in Statistical Programming}},
  year = {2025},
  eprint = {2502.13117},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13117},
}
Figure