Perish or Flourish? A Holistic Evaluation of Large Language Models for Code Generation in Functional Programming

Nguyet-Anh H. Lang, Eric Lang, Thanh Le-Cong, Bach Le, Quyet-Thang HuynharXiv 2026 · arXiv

Abstract

Functional programming provides strong foundations for developing reliable and secure software systems, yet its adoption remains not widespread due to the steep learning curve. Recent advances in Large Language Models (LLMs) for code generation present new opportunities to lower these barriers. However, extensive evaluations of LLMs largely focus on imperative programming languages, and their capabilities in functional programming languages (FP) remain underexplored. To address this gap, we introduce FPEval, a holistic evaluation framework built on FPBench, a new benchmark of 721 programming tasks across three difficulty levels on three mainstream FP languages: Haskell, Ocaml and Scala. FPEval provides compehensive evaluation infrastructures with both test validations with comprehensive test suites and static analysis tools to assess both functional correctness and code style and maintainability. Using this framework, we evaluate state-of-the-art LLMs, including GPT-3.5, GPT-4o, and GPT-5, for code generation in functional programming languages and Java as an imperative baseline. Our results demonstrate that LLM performance in functional programming improves substantially with model advancement; however, error rates remain significantly higher in purely functional languages (Haskell and OCaml) than in hybrid (Scala) or imperative (Java) languages. Moreover, LLMs frequently generate non-idiomatic functional code that follows imperative patterns, raising concerns about code style and long-term maintainability. Finally, we show that LLMs can partially self-repair both correctness and quality issues when provided with static analysis feedback and hand-crafted instructions for common types of issues.

Introduction

Functional Programming (FP) languages like Haskell, OCaml, and Scala offer powerful tools for building secure, concurrent, and bug-resistant software. However, their steep learning curve—requiring a shift from step-by-step commands to high-level mathematical abstractions—limits their adoption. While LLMs have revolutionized coding in imperative languages like Python and Java, their proficiency in the functional paradigm remains underexplored. Most existing benchmarks are biased toward imperative patterns, leaving a gap in our understanding of whether LLMs can truly "think" functionally or if they are simply forcing imperative logic into functional syntax.

Methodology

To address this, we developed FPEval, a holistic evaluation framework designed to move beyond simple "pass/fail" testing. The framework utilizes FPBench, a new benchmark consisting of 721 programming tasks sourced from LeetCode (2021–2025), categorized into Easy (184), Medium (346), and Hard (191) levels.

FPEval employs a dual-track evaluation strategy:

  1. Functional Correctness: Assessed via automated transpilation of Python signatures into FP starters and testing against both public and synthesized private test suites.
  2. Code Quality & Idiomaticity: Integration of industry-standard static analysis tools (HLint, OCamlFormat, and Scalastyle) to detect "imperative bias"—the use of non-functional patterns like mutable variables in a functional context.

Evaluation Results

Our evaluation reveals a persistent "paradigm gap" between pure FP and imperative languages. While model performance scales with complexity, there is a significant chasm in success rates. For example, GPT-5 achieved a massive improvement in correctness, solving roughly 3x more tasks than GPT-3.5. However, even this advanced model struggles with the strictness of functional compilers compared to its performance in Java.

Performance metrics of GPT models across functional and imperative languages.

As shown in the data, GPT-3.5 failed to compile its OCaml solutions 43.14% of the time, highlighting a fragility in syntactic understanding that persists, albeit at lower rates, in newer models. While GPT-5 reached a 42.34% pass rate in Haskell, this still lags significantly behind its 61.14% pass rate in Java.

The Quality-Correctness Trade-off

A striking finding of this study is the "Reward Hacking" phenomenon. As models become more capable of solving the underlying logic (increasing pass rates), their code quality often declines. In Scala, for instance, the "clean code" rate—representing code that passes all static analysis checks—dropped significantly as models advanced.

Idiomaticity and code quality rates across languages and models.

In Scala, the clean code rate dropped from 77% for GPT-4o to just 57% for GPT-5. This suggests that to solve more complex problems, models frequently resort to imperative "hacks" (like using var and return) rather than employing idiomatic functional constructs like recursion or higher-order functions.

Conclusion

While LLMs are rapidly improving their ability to generate functional code, they do not yet "think" in the functional paradigm as effectively as they do in imperative ones. The tendency to prioritize logical correctness over idiomatic style suggests that current training sets may lack sufficient high-quality functional examples. However, our findings on self-repair indicate that providing models with specific feedback from static analysis tools can significantly bridge this gap, improving both the correctness and the maintainability of the generated code.

Summary updated

Abstract

Functional programming provides strong foundations for developing reliable and secure software systems, yet its adoption remains not widespread due to the steep learning curve. Recent advances in Large Language Models (LLMs) for code generation present new opportunities to lower these barriers. However, extensive evaluations of LLMs largely focus on imperative programming languages, and their capabilities in functional programming languages (FP) remain underexplored. To address this gap, we introduce FPEval, a holistic evaluation framework built on FPBench, a new benchmark of 721 programming tasks across three difficulty levels on three mainstream FP languages: Haskell, Ocaml and Scala. FPEval provides compehensive evaluation infrastructures with both test validations with comprehensive test suites and static analysis tools to assess both functional correctness and code style and maintainability. Using this framework, we evaluate state-of-the-art LLMs, including GPT-3.5, GPT-4o, and GPT-5, for code generation in functional programming languages and Java as an imperative baseline. Our results demonstrate that LLM performance in functional programming improves substantially with model advancement; however, error rates remain significantly higher in purely functional languages (Haskell and OCaml) than in hybrid (Scala) or imperative (Java) languages. Moreover, LLMs frequently generate non-idiomatic functional code that follows imperative patterns, raising concerns about code style and long-term maintainability. Finally, we show that LLMs can partially self-repair both correctness and quality issues when provided with static analysis feedback and hand-crafted instructions for common types of issues.
BibTeX
@misc{lang2026perish,
  author = {Nguyet-Anh H. Lang and Eric Lang and Thanh Le-Cong and Bach Le and Quyet-Thang Huynh},
  title = {{Perish or Flourish? A Holistic Evaluation of Large Language Models for Code Generation in Functional Programming}},
  year = {2026},
  eprint = {2601.02060},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.02060},
  howpublished = {arXiv},
}
Figure