TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation

Zhihao Gong, Zeyu Sun, Dong Huang, Qingyuan Liang, Jie M. Zhang, Dan HaoarXiv 2025 · DOI · arXiv

Abstract

While Large Language Models (LLMs) have made significant strides in automated code translation, evaluation has primarily focused on functional correctness. This paper introduces TRACY, a comprehensive benchmark designed to evaluate the execution efficiency of translated code. TRACY reveals that functionally correct translations can suffer from massive performance degradations—up to 600x—due to algorithmic hallucinations and a lack of target-language idiomatic awareness.

Introduction: The Efficiency Blind Spot

In the world of automated code translation (e.g., moving a legacy C++ codebase to Python), "correctness" has long been the only metric that matters. If the output of an LLM passes a few unit tests, it is deemed a success. However, functional equivalence does not guarantee performance equivalence.

A translation can be logically sound but computationally disastrous—replacing an efficient bitwise operation with a slow loop or using immutable string concatenation in a hot loop. Existing benchmarks typically rely on "small tests" that fail to trigger these performance bottlenecks, allowing inefficient code to pass undetected. TRACY aims to turn code translation evaluation from a simple "pass/fail" correctness check into a rigorous performance gauntlet.

Example of a 600x performance gap in C++ to Python translation where a model fails to use efficient bitwise logic.

Methodology: Building an Efficiency-Oriented Benchmark

The researchers developed an LLM-driven, two-stage pipeline to build the TRACY benchmark:

  1. Stress Test Generation: Instead of relying on simple inputs, they used GPT-4o to synthesize "synthesizers"—Python scripts that programmatically generate massive, complex test inputs. These inputs are designed to amplify performance differences that remain hidden during trivial execution.
  2. Efficiency-Oriented Task Pruning: Tasks were filtered for feasibility (at least one LLM must solve it), impactfulness (non-trivial execution costs), and diversity (significant variance in performance across models).
The TRACY pipeline for automated stress test generation and task pruning.

To quantify performance, the benchmark uses the Beyond score, which normalizes a model's execution time and memory usage against a distribution of verified reference translations.

The TRACY Benchmark Dataset

TRACY is built upon the foundation of Transcoder-Test, expanding it into a robust efficiency-critical dataset:

  • Tasks: 1,011 efficiency-critical tasks spanning six language pairs (C++, Java, and Python).
  • References: Each task includes an average of 22.1 verified reference translations.
  • Tests: Each task includes 10 computationally demanding stress tests alongside standard unit tests.
  • Evaluation Scope: The authors tested 26 representative LLMs, including proprietary giants (GPT-4o, Claude-4, O3) and open-source contenders (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder).

Key Insights and Results

The Correctness-Efficiency Disconnect

High accuracy does not equal high performance. Claude-4-think, the leader in functional correctness (95% pass rate), plummeted to 8th place when ranked by execution time. It was outperformed by several smaller, open-source models like QC-14B-Inst, which showed remarkable efficiency.

Root Causes of Inefficiency

A manual analysis of 236 inefficient cases identified two primary culprits:

  • Idiomatic/Library Usage Errors (61.9% of cases): Models often fail to adopt target-language idioms. For example, translating Python's string concatenation literally into Java (which uses immutable strings) leads to $O(N^2)$ complexity, whereas an idiomatic Java translation would use StringBuilder for $O(N)$ performance.
  • Algorithmic Deviations: Some models "hallucinate" simpler but less efficient logic. In one case, a model replaced an optimized bitwise GCD algorithm with a basic subtraction-based one, leading to a 600x performance gap under stress.

Asymmetric Efficiency

Models are generally better at producing efficient code when translating Java → Python than the reverse. Conversely, memory efficiency is much higher when targeting Java from C++, likely due to Java’s more direct mapping of primitive types compared to Python’s "everything is an object" overhead.

Conclusion

TRACY highlights a critical need for efficiency-aware evaluation in code translation. While the benchmark currently focuses on C++, Java, and Python, future work should expand to languages like Rust or Go where memory management is even more critical. The findings suggest that future model development should focus on "Efficiency-Aware Fine-tuning" to help models prioritize optimal algorithmic complexity during the translation process.

Summary updated

BibTeX
@misc{gong2025tracy,
  author = {Zhihao Gong and Zeyu Sun and Dong Huang and Qingyuan Liang and Jie M. Zhang and Dan Hao},
  title = {{TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation}},
  year = {2025},
  eprint = {2508.11468},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.11468},
}
Figure