Revisiting the Role of Natural Language Code Comments in Code Translation

Monika Gupta, Ajay Kumar Meena, Anamitra Roy Choudhury, Vijay Arya, Srikanta Bedathur

Abstract

The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than 80,000 translations, with and without comments, of 1100+ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation.

Introduction

Automated code translation is a critical task for modernizing legacy systems and migrating software across platforms. While Large Language Models (LLMs) like CodeLlama and DeepSeekCoder have shown remarkable capabilities, they often struggle with complex logic, leading to functional errors or "hallucinations." Researchers from IIT Delhi and IBM Research India recently investigated whether natural language code comments—often stripped from standard benchmarks—could serve as the "secret sauce" to improve translation accuracy.

The COMMENTRA Framework

The core contribution of this research is COMMENTRA, an iterative translation framework designed to leverage the descriptive power of comments only when necessary.

The COMMENTRA iterative translation framework.

The framework operates in four stages:

  1. Initial Attempt: The translating LLM attempts to convert raw, uncommented source code into the target language.
  2. Verification: The resulting code is validated using compilers and test cases.
  3. Iterative Feedback: If the initial translation fails, a "Commenting LLM" (e.g., GPT-4o-mini) generates high-level natural language comments for the original source code.
  4. Re-translation: The translating LLM makes a second attempt, now guided by the newly injected comments.

This "lazy" approach ensures computational efficiency by only invoking expensive commenting models for difficult samples.

Experimental Methodology

The study involved a massive empirical evaluation of over 80,000 translations across five programming languages: C, C++, Go, Java, and Python. The researchers utilized several benchmarks, including AVATAR and CodeNet, alongside a newly curated dataset, CJBench, which contains 200 Java programs with high-quality, human-written comments to study the impact of comment intent.

Key Insights: Intent and Language

The study revealed several critical nuances regarding how comments affect LLMs:

  • Descriptive vs. Line-by-Line: High-level comments explaining the overall purpose of a block are significantly more effective than line-by-line explanations. Granular comments can sometimes introduce "noise" or lead to hallucinations, such as a model assuming a fixed-size array based on a vague keyword.
  • The Dominance of English: English comments provided the most consistent performance boost, even for translations between non-English contexts, reflecting the English-centric nature of LLM training data.
  • Targeted Application: Indiscriminate commenting can occasionally degrade performance. The most effective strategy is using comments as a targeted debugging aid for failed translations.

Results and Discussion

The results demonstrate that adding comments can double the number of successful translations in certain configurations. The COMMENTRA framework achieved average performance gains between 23% and 71% after just two iterations.

Sankey diagrams showing the transition of translation outcomes with and without GPT-generated comments across different LLMs.

As shown in the Sankey diagrams, the introduction of comments significantly shifts outcomes from "CompileFail" and "TestCaseFail" toward "Success" across various models, including DeepSeek and GPT. However, model sensitivity varies; while larger models benefit greatly, smaller models like StarCoder can sometimes be confused by the additional natural language context.

Conclusion

This study provides the first comprehensive characterization of the impact of code comments on automated translation. By demonstrating that high-level intent-driven comments can significantly boost LLM performance, the authors highlight a path toward more robust automated migration tools. Future work will focus on scaling COMMENTRA to repository-level projects with complex inter-file dependencies and developing specialized models for generating translation-optimized comments.

Summary updated

Abstract

The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than $80,000$ translations, with and without comments, of $1100+$ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation. To the best of our knowledge, our study is the first in terms of its comprehensiveness, scale, and language coverage on how to improve code translation accuracy using code comments.
BibTeX
@inproceedings{gupta2026revisiting,
  author = {Monika Gupta and Ajay Kumar Meena and Anamitra Roy Choudhury and Vijay Arya and Srikanta Bedathur},
  title = {{Revisiting the Role of Natural Language Code Comments in Code Translation}},
  booktitle = {FASE},
  year = {2026},
  doi = {10.1007/978-3-032-22774-4\_2},
}
Figure