LLM-Assisted Translation of Legacy FORTRAN Codes to C++: A Cross-Platform Study

Nishath Rajiv Ranasinghe, Shawn M. Jones, Michal Kucer, Ayan Biswas, Daniel O’Malley, Alexander Most, Selma Liliane Wanna, Ajay SreekumarNAACL 2025

TL;DR

This paper explores the effectiveness of Large Language Models (LLMs) in translating legacy Fortran scientific code to C++. They developed a comprehensive, platform-agnostic evaluation workflow that assesses translation quality through compilation success, similarity to human translations, and output equivalence, offering valuable insights into LLM capabilities for modernizing scientific software.

What Problem Does This Paper Address?

A significant portion of high-performance computing (HPC) scientific software is written in legacy languages like Fortran. While Fortran has been instrumental in scientific discovery, it struggles to leverage modern hardware and faces challenges with memory management. Maintaining and updating these codebases is difficult due to a lack of human resources and the aging nature of the language. Translating this legacy code to modern languages like C++ is crucial for continued development and performance optimization. However, existing LLM-based code translation methods, especially for Fortran to C++, have not been thoroughly evaluated for their practical usability, particularly concerning compilation accuracy and functional equivalence. This paper aims to fill that gap by systematically assessing LLM performance on this specific translation task across different computational platforms.

Methods

The authors propose an LLM-independent and platform-independent evaluation workflow. This workflow begins by comparing LLM-generated C++ code against a human-translated "ground truth" using the CodeBLEU metric, which measures similarity based on n-grams, abstract syntax trees, and data flow. Next, they assess the compilation accuracy of the translated C++ code using the g++ compiler. If a translation compiles, they then evaluate its output similarity by comparing the execution output of the original Fortran program with the translated C++ program. This is done by tokenizing outputs and using a Jaro-Winkler score to measure string similarity, with a focus on handling floating-point precision differences. Compile errors are categorized to understand common failure modes. The study also accounts for the stochastic nature of LLMs by running multiple translation attempts for each code snippet.

Datasets

The study utilized two datasets for evaluation:

  • Rosetta Code: A web scraper was used to collect 243 pairs of Fortran and corresponding C++ examples from the Rosetta Code repository. Only examples with at least one Fortran and one C++ implementation per programming task were retained.
  • DataRaceBench (DRB) benchmark: 101 examples were sourced from the DRB benchmark, specifically from the OpenMP Fortran to C++ dataset.

These datasets were combined to create a total of 344 computer programs of varying complexity, ensuring that the ground truth Fortran and C++ programs could be compiled.

Key Insights and Ideas

  • Larger models generally perform better: The study consistently shows that LLMs with a higher number of parameters tend to produce better results, whether measured by CodeBLEU similarity, compilation accuracy, or output similarity. This suggests that scaling up LLMs is a promising direction for improving code translation.
  • Platform matters, but LLM choice is key: While there are differences in performance between the vLLM and SambaNova platforms, the choice of LLM often has a more significant impact on translation quality.
  • Compile errors reveal platform differences: LLMs served via vLLM predominantly produced linker errors (indicating missing library includes), while those on SambaNova Cloud more frequently generated syntax errors. This highlights how the execution environment can influence the types of failures encountered.
  • No LLM is perfect: Despite advancements, no LLM evaluated in this study was able to achieve perfect translation across all metrics. This underscores the continued need for human oversight in the code translation process.
  • Fortran to C++ is challenging but feasible: The study demonstrates that LLMs can perform this translation with varying degrees of success, providing a foundation for future agentic workflows.

Key Results

  • Similarity to Human Translation (CodeBLEU): Mistral Large on vLLM and Llama 3.1 405B on SambaNova Cloud showed the best performance in matching human translations. However, Llama 3.1 70B on vLLM had the highest bias (least similarity).
  • Compilation Accuracy: Generally, LLMs with more parameters achieved higher compilation accuracy. For vLLM, Llama 3.1 70B and Llama 3.3 70B performed comparably. On SambaNova, Llama 3.1 405B and Llama 3.3 70B showed similar performance.
  • Output Similarity (Jaro-Winkler): Mistral Large on vLLM and Llama 3.1 405B on SambaNova Cloud outperformed other models in producing outputs that matched the original Fortran programs. Mistral Large exhibited a tighter distribution of similar outputs.

Limitations and Future Work

The authors acknowledge several limitations:

  • Workflow not yet a standalone tool: The evaluation workflow is not packaged as an automated tool, which could hinder its adoption by researchers.
  • No agentic error correction explored: The study did not incorporate agentic workflows to improve compilation accuracy by feeding compiler errors back to the LLM. Initial efforts in this direction showed promise and will be pursued.
  • Limited code complexity: The study could benefit from evaluating more complex and extensive Fortran codebases.
  • Focus on open-weight LLMs: Proprietary models like GPT and Gemini, as well as older source-to-source translation tools, were not included in the comparison.
  • No few-shot learning or RAG: The study did not explore techniques like few-shot learning or Retrieval Augmented Generation (RAG) for improving translation.

Future work could involve developing the workflow into a user-friendly tool, implementing agentic error correction, expanding the dataset to include more complex code, and comparing open-weight LLMs with proprietary models and established translation tools.

Summary updated

BibTeX
@inproceedings{ranasinghe2025llm,
  author = {Nishath Rajiv Ranasinghe and Shawn M. Jones and Michal Kucer and Ayan Biswas and Daniel O’Malley and Alexander Most and Selma Liliane Wanna and Ajay Sreekumar},
  title = {{LLM-Assisted Translation of Legacy FORTRAN Codes to C++: A Cross-Platform Study}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure