Revisiting the Role of Natural Language Code Comments in Code Translation

Monika Gupta, Ajay Meena, Anamitra Roy Choudhury, Vijay Arya, Srikanta Bedathur

This paper, recently published by researchers from IIT Delhi and IBM Research India, explores a simple but often overlooked question in AI-driven software engineering: Do code comments actually help Large Language Models (LLMs) translate code between programming languages?

While we often strip comments for "clean" benchmarks, this study suggests that natural language context is the secret sauce that can significantly boost—or occasionally break—automated translation.

TL;DR

Researchers demonstrate that natural language comments can significantly improve LLM-based code translation, sometimes doubling performance. By introducing COMMENTRA, an iterative framework that injects high-level descriptive comments only when a translation fails, they achieve performance gains of up to 71% across major programming languages.

What Problem Does This Paper Address?

Automated code translation (e.g., converting Java to Python) is a "holy grail" for legacy system migration. While LLMs like CodeLlama and DeepSeekCoder are powerful, they still struggle with logic errors and "hallucinations" in complex programs.

The authors identify a major gap: most existing benchmarks use uncommented code. However, LLMs are trained on massive repositories like GitHub, where code and comments coexist. The researchers set out to characterize how comment intent (what they say), density (how many there are), language (English vs. others), and location affect an LLM's ability to translate code accurately.

Methods

The authors conducted a massive empirical study involving over 80,000 translations across five languages: C, C++, Go, Java, and Python.

The core contribution is COMMENTRA, an iterative translation framework:

  1. Initial Attempt: The model tries to translate the raw, uncommented code.
  2. Verification: The output is run against test cases and compilers.
  3. Iterative Feedback: If the translation fails, a "Commenting LLM" (like GPT-4o-mini) generates natural language comments for the source code.
  4. Re-translation: The model attempts the translation again, now "primed" with the new comments.

This approach is "lazy" and cost-effective—it only uses expensive LLM calls for commenting when the simple approach fails.

Datasets

The study utilized several key benchmarks:

  • AVATAR & CodeNet: Standard benchmarks for Java, Python, C, C++, and Go.
  • CJBench (New): A manually curated dataset of 200 Java programs with high-quality, author-written comments, used specifically to study the impact of "comment intent."
  • CodeTransOcean: A multilingual benchmark used for further validation.

Key Insights and Ideas

  • Descriptive > Line-by-Line: Comments that explain the overall purpose of a function are far more helpful than line-by-line explanations. In fact, line-by-line comments can sometimes "trick" the model.
  • The "Grid" Hallucination: In one fascinating example, a comment mentioning a "grid" led the translating LLM to declare a fixed-size 2D array (grid[100][100]) that wasn't in the original code, causing a memory fault. This highlights that comments can act as "noise" if they are too specific or poorly phrased.
  • English is the Lingua Franca: Even when translating between non-English contexts, English comments provided the best performance boost, likely due to the English-centric nature of LLM training data.
  • Indiscriminate Commenting Hurts: Adding comments to every single file can actually decrease performance. The "sweet spot" is using comments as a targeted debugging aid when the model is stuck.

Key Results

  • Performance Doubling: In several configurations, adding comments doubled the number of successful translations compared to the uncommented baseline.
  • COMMENTRA Gains: The iterative framework showed average performance gains between 23% and 71% after two iterations (using DeepSeek and GPT as commenting models).
  • Model Sensitivity: While models like GPT-4o-mini and DeepSeek were excellent at using comments to improve, older or smaller models (like StarCoder) were sometimes more easily confused by the extra text.

Limitations and Future Work

The authors acknowledge that their study focused on standalone programs. Real-world software involves repository-scale complexities, inter-file dependencies, and massive codebases that might exceed an LLM's "context window."

Future work includes:

  • Repository-Scale Translation: Testing COMMENTRA on large, multi-file projects.
  • Comment Quality Metrics: Developing better ways to measure the "helpfulness" of a comment before passing it to a translator.
  • Intent-Driven Generation: Training specialized models to generate comments specifically designed to aid translation rather than human readability.

Summary updated

Abstract

The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than $80,000$ translations, with and without comments, of $1100+$ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation. To the best of our knowledge, our study is the first in terms of its comprehensiveness, scale, and language coverage on how to improve code translation accuracy using code comments.
BibTeX
@inproceedings{gupta2026revisiting,
  author = {Monika Gupta and Ajay Meena and Anamitra Roy Choudhury and Vijay Arya and Srikanta Bedathur},
  title = {{Revisiting the Role of Natural Language Code Comments in Code Translation}},
  booktitle = {FASE},
  year = {2026},
  doi = {10.1007/978-3-032-22774-4\_2},
}

Loading related content...

Figure