Abstract
While Large Language Models (LLMs) have made significant strides in automated code translation, evaluation has primarily focused on functional correctness. This paper introduces TRACY, a comprehensive benchmark designed to evaluate the execution efficiency of translated code. TRACY reveals that functionally correct translations can suffer from massive performance degradations—up to 600x—due to algorithmic hallucinations and a lack of target-language idiomatic awareness.
Introduction: The Efficiency Blind Spot
In the world of automated code translation (e.g., moving a legacy C++ codebase to Python), "correctness" has long been the only metric that matters. If the output of an LLM passes a few unit tests, it is deemed a success. However, functional equivalence does not guarantee performance equivalence.
A translation can be logically sound but computationally disastrous—replacing an efficient bitwise operation with a slow loop or using immutable string concatenation in a hot loop. Existing benchmarks typically rely on "small tests" that fail to trigger these performance bottlenecks, allowing inefficient code to pass undetected. TRACY aims to turn code translation evaluation from a simple "pass/fail" correctness check into a rigorous performance gauntlet.
Methodology: Building an Efficiency-Oriented Benchmark
The researchers developed an LLM-driven, two-stage pipeline to build the TRACY benchmark:
- Stress Test Generation: Instead of relying on simple inputs, they used GPT-4o to synthesize "synthesizers"—Python scripts that programmatically generate massive, complex test inputs. These inputs are designed to amplify performance differences that remain hidden during trivial execution.
- Efficiency-Oriented Task Pruning: Tasks were filtered for feasibility (at least one LLM must solve it), impactfulness (non-trivial execution costs), and diversity (significant variance in performance across models).
To quantify performance, the benchmark uses the Beyond score, which normalizes a model's execution time and memory usage against a distribution of verified reference translations.
The TRACY Benchmark Dataset
TRACY is built upon the foundation of Transcoder-Test, expanding it into a robust efficiency-critical dataset:
- Tasks: 1,011 efficiency-critical tasks spanning six language pairs (C++, Java, and Python).
- References: Each task includes an average of 22.1 verified reference translations.
- Tests: Each task includes 10 computationally demanding stress tests alongside standard unit tests.
- Evaluation Scope: The authors tested 26 representative LLMs, including proprietary giants (GPT-4o, Claude-4, O3) and open-source contenders (CodeLlama, Qwen2.5-Coder, DeepSeek-Coder).
Key Insights and Results
The Correctness-Efficiency Disconnect
High accuracy does not equal high performance. Claude-4-think, the leader in functional correctness (95% pass rate), plummeted to 8th place when ranked by execution time. It was outperformed by several smaller, open-source models like QC-14B-Inst, which showed remarkable efficiency.
Root Causes of Inefficiency
A manual analysis of 236 inefficient cases identified two primary culprits:
- Idiomatic/Library Usage Errors (61.9% of cases): Models often fail to adopt target-language idioms. For example, translating Python's string concatenation literally into Java (which uses immutable strings) leads to $O(N^2)$ complexity, whereas an idiomatic Java translation would use
StringBuilderfor $O(N)$ performance. - Algorithmic Deviations: Some models "hallucinate" simpler but less efficient logic. In one case, a model replaced an optimized bitwise GCD algorithm with a basic subtraction-based one, leading to a 600x performance gap under stress.
Asymmetric Efficiency
Models are generally better at producing efficient code when translating Java → Python than the reverse. Conversely, memory efficiency is much higher when targeting Java from C++, likely due to Java’s more direct mapping of primitive types compared to Python’s "everything is an object" overhead.
Conclusion
TRACY highlights a critical need for efficiency-aware evaluation in code translation. While the benchmark currently focuses on C++, Java, and Python, future work should expand to languages like Rust or Go where memory management is even more critical. The findings suggest that future model development should focus on "Efficiency-Aware Fine-tuning" to help models prioritize optimal algorithmic complexity during the translation process.
Summary updated