This paper, recently published by researchers from IIT Delhi and IBM Research India, explores a simple but often overlooked question in AI-driven software engineering: Do code comments actually help Large Language Models (LLMs) translate code between programming languages?
While we often strip comments for "clean" benchmarks, this study suggests that natural language context is the secret sauce that can significantly boost—or occasionally break—automated translation.
TL;DR
Researchers demonstrate that natural language comments can significantly improve LLM-based code translation, sometimes doubling performance. By introducing COMMENTRA, an iterative framework that injects high-level descriptive comments only when a translation fails, they achieve performance gains of up to 71% across major programming languages.
What Problem Does This Paper Address?
Automated code translation (e.g., converting Java to Python) is a "holy grail" for legacy system migration. While LLMs like CodeLlama and DeepSeekCoder are powerful, they still struggle with logic errors and "hallucinations" in complex programs.
The authors identify a major gap: most existing benchmarks use uncommented code. However, LLMs are trained on massive repositories like GitHub, where code and comments coexist. The researchers set out to characterize how comment intent (what they say), density (how many there are), language (English vs. others), and location affect an LLM's ability to translate code accurately.
Methods
The authors conducted a massive empirical study involving over 80,000 translations across five languages: C, C++, Go, Java, and Python.
The core contribution is COMMENTRA, an iterative translation framework:
- Initial Attempt: The model tries to translate the raw, uncommented code.
- Verification: The output is run against test cases and compilers.
- Iterative Feedback: If the translation fails, a "Commenting LLM" (like GPT-4o-mini) generates natural language comments for the source code.
- Re-translation: The model attempts the translation again, now "primed" with the new comments.
This approach is "lazy" and cost-effective—it only uses expensive LLM calls for commenting when the simple approach fails.
Datasets
The study utilized several key benchmarks:
- AVATAR & CodeNet: Standard benchmarks for Java, Python, C, C++, and Go.
- CJBench (New): A manually curated dataset of 200 Java programs with high-quality, author-written comments, used specifically to study the impact of "comment intent."
- CodeTransOcean: A multilingual benchmark used for further validation.
Key Insights and Ideas
- Descriptive > Line-by-Line: Comments that explain the overall purpose of a function are far more helpful than line-by-line explanations. In fact, line-by-line comments can sometimes "trick" the model.
- The "Grid" Hallucination: In one fascinating example, a comment mentioning a "grid" led the translating LLM to declare a fixed-size 2D array (
grid[100][100]) that wasn't in the original code, causing a memory fault. This highlights that comments can act as "noise" if they are too specific or poorly phrased. - English is the Lingua Franca: Even when translating between non-English contexts, English comments provided the best performance boost, likely due to the English-centric nature of LLM training data.
- Indiscriminate Commenting Hurts: Adding comments to every single file can actually decrease performance. The "sweet spot" is using comments as a targeted debugging aid when the model is stuck.
Key Results
- Performance Doubling: In several configurations, adding comments doubled the number of successful translations compared to the uncommented baseline.
- COMMENTRA Gains: The iterative framework showed average performance gains between 23% and 71% after two iterations (using DeepSeek and GPT as commenting models).
- Model Sensitivity: While models like GPT-4o-mini and DeepSeek were excellent at using comments to improve, older or smaller models (like StarCoder) were sometimes more easily confused by the extra text.
Limitations and Future Work
The authors acknowledge that their study focused on standalone programs. Real-world software involves repository-scale complexities, inter-file dependencies, and massive codebases that might exceed an LLM's "context window."
Future work includes:
- Repository-Scale Translation: Testing COMMENTRA on large, multi-file projects.
- Comment Quality Metrics: Developing better ways to measure the "helpfulness" of a comment before passing it to a translator.
- Intent-Driven Generation: Training specialized models to generate comments specifically designed to aid translation rather than human readability.
Summary updated