Many developers write code comments in their own language, and those comments are rarely pure Chinese or pure Indonesian. They are mixed text: native grammar carrying English identifiers, API names and jargon. Nearly every system that generates comments automatically is trained and evaluated on English docstrings. The usual workaround for other languages is to generate an English comment and machine-translate it. This proposal asks when that workaround breaks, and whether a compact code model pre-trained on target-language developer text fixes the break.
The key move is a measurement. We use translation divergence, meaning how much a comment changes when it is translated to English and back, to sort real native comments by how "untranslatable" they are. Our prediction is that a localized model's advantage over the translate-afterwards pipeline is small on comments that survive round-trip translation and large on comments that do not. If that pattern holds, it tells practitioners which documentation actually needs localized models. If it fails, the cheap pipeline is good enough, and we will say so.
Core Claim
The monolingual pre-training advantage reported for general text should also appear in non-English code documentation, and it should be concentrated in a predictable subset. That subset consists of comments whose meaning or register is lost under round-trip translation, typically idiomatic phrasing and natural code-switching. We test three claims together.
First, a 125–220M-parameter code-text model extended to the target language beats two alternatives on native-comment generation for Chinese and Indonesian: a multilingual code model fine-tuned on the same data, and an English-summarize-then-translate pipeline. The extension has two parts: continued pre-training on target-language developer text, and a tokenizer extended for that language.
Second, the localized model's margin over the pipeline grows monotonically with the round-trip divergence of the reference comment.
Third, keeping organic code-switching in the pre-training corpus matters. Normalizing it away should reproduce the harm that CroCoSum observed.
Grounding in Prior Work
Three groups of work motivate the proposal. A fourth line of work supplies its methods.
The divergence lens comes from multiword-expression research.
- Detecting Non-compositional MWE Components using Wiktionary compares the translation of a whole expression with the translations of its parts and flags mismatches as non-compositional.
- Helsinki-NLP at SemEval-2022 Task 2 uses back-translation features for idiomaticity detection. It reports F1 of 0.752 for English, 0.694 for Portuguese, and only 0.499 for Galician, which shows that noisy translation engines obscure the signal.
- BERT-based Idiom Identification using Language Translation and Word Cohesion turns round-trip METEOR drift through Hindi into a training loss and reaches 91.51% sequence accuracy on MAGPIE.
All three papers use divergence to detect idioms in general prose. We use it differently, as a stratification variable that predicts where a translation pipeline will fail on technical documentation. The Galician result constrains our design: we must control for translation-engine noise and cannot treat raw divergence as pure signal.
The localized-model hypothesis comes from three studies. IndoNLG reports that IndoBART, at roughly 20% of mBART-Large's size, gets the best Liputan6 summarization scores. SeaLLMs shows that extending the tokenizer cuts Thai compression from 4.29 to 1.57 and lifts translated-GSM8K Thai from ChatGPT-3.5's 35.8 to 62.0. NLEBench+NorGLM reports similar wins for Norwegian. None of these studies touches code.
The closest bridge is CroCoSum, which studies English-to-Chinese technology-news summarization. In its summaries, more than 92% are code-switched. It reports three findings that matter here:
- An end-to-end mBART-50 model (34.35 ROUGE-L) beats both pipelines and zero-shot GPT-3.5 (31.17).
- Model outputs show 30% over-switching and 43% erroneous switches.
- Pre-training on standard cross-lingual corpora hurt performance.
CroCoSum leaves three things open. It never generates from code, never tests a localized model, and never asks which instances drive the pipeline's loss. We take up all three.
On the documentation side, several papers supply method and constraints.
- CodeXGLUE covers multilingual documentation only as text-to-text translation of Microsoft Docs, not as code-to-comment generation.
- DocChecker shows that a 124M UniXcoder with targeted training beats CodeLlama-34B by about 10% accuracy on code-comment inconsistency. This is evidence that compact specialized models are a fair choice, but the work is English-only.
- The Vault supplies the quality filter we reuse for mining comments; its neural classifier reaches AUC 0.89.
- EditSum supplies the project-level split we adopt.
- CodeSearchNet Challenge acts as a warning. It shows that docstring-matching scores (MRR 0.7011 for self-attention) can reverse on real queries, where self-attention's NDCG of 0.493 falls below neural bag-of-words at 0.574. This is why we do not rely on BLEU alone.
Research Program
Building the benchmark
The first deliverable is a benchmark of code paired with native-language comments in Chinese and Indonesian. We chose these two languages because Chinese developer comments are abundant and CroCoSum documents Chinese technical code-switching, while IndoNLG supplies Indonesian pre-training resources.
The pipeline runs as follows:
- Mine Python and Java functions from public repositories whose docstrings or block comments are identified as the target language.
- Apply The Vault's quality filter.
- Discard placeholder comments.
- Split by project, so that no repository contributes to both training and test.
A test subset of about 2,000 functions per language will additionally be rated by native-speaking developers.
Defining the divergence score
For each reference comment $c$, we first mask code identifiers and API names. We then translate the masked comment to English and back with two independent machine translation systems, $T_1$ and $T_2$, and compute
$$d(c) = 1 - \tfrac{1}{2}\left[\mathrm{chrF}(c, T_1(c)) + \mathrm{chrF}(c, T_2(c))\right],$$
where $T_k(c)$ is the round trip through system $k$ and chrF is character n-gram F-score. Character-level scoring suits Chinese and Indonesian morphology.
Two controls separate genuine non-translatability from engine noise:
- Engine disagreement. We record the disagreement between $T_1$ and $T_2$. If the score mainly reflects one weak engine, the two systems should disagree on the high-divergence items.
- External validation. We check that $d$ separates idiomatic from literal items on SemEval-2022 Task 2 (English and Portuguese) better than chance, and that it correlates with developer ratings of "would a native developer phrase it this way."
The validation target is only a meaningful signal. We do not need to match Helsinki-NLP's supervised F1.
Building the localized model
We start from a compact code-text encoder-decoder and make two changes:
- Tokenizer extension. Following SeaLLMs, we add target-language vocabulary.
- Continued pre-training. We train on target-language developer text: forum posts, issue threads, READMEs and comments, with code-switching left intact.
To guard against forgetting, which SeaLLMs reports, a share of English code-text data is mixed into the pre-training stream. English CodeXGLUE summarization is monitored throughout.
Baselines
Naive reference points. The do-nothing floor is a retrieval baseline in the style of EditSum's prototype step. It returns the comment of the most lexically similar training function, using Lucene-style matching on the same project-split training set. Any learned model that fails to beat this floor is only memorizing comment templates. A second reference point is the off-the-shelf multilingual code model fine-tuned directly on native comments, with no further adaptation. This isolates whether anything beyond ordinary fine-tuning is needed.
Pipeline baseline. The strongest practical competitor is the English pipeline. A CodeT5-class summarizer trained on The Vault, for which The Vault reports 18.83 BLEU-4 on the CodeSearchNet test set, generates an English comment, and the stronger of our two translation systems then translates it. This pipeline is what a team would actually deploy, and it costs no new pre-training.
To keep the comparison fair, the English summarizer has the same parameter count as our model. The pipeline is also allowed to fine-tune on English back-translations of our native training comments, so that it sees the same supervision signal.
The pipeline wins if it is within the native developers' rating noise on the full test set and on the top divergence quintile. Beating it means a statistically significant advantage on human-rated adequacy and naturalness that is concentrated in the high-divergence quintiles.
For calibration only: DocChecker reports 33.64 BLEU-4 on English CodeXGLUE summarization. No reported result was found on Lacuna for code summarization into Chinese or Indonesian, so no external number exists to beat.
Large-model baseline. We prompt a large general model zero-shot and few-shot. CroCoSum's 31.17 versus 34.35 ROUGE-L gap suggests this baseline will lose, but code may differ from news.
Ablations. Four ablations isolate the key ingredients, each trained on matched tokens with matched tuning budgets:
- Corpus content. Continued pre-training on an equal amount of English code-text instead of target-language text. This rules out "just more training."
- Tokenizer. The localized model without the tokenizer extension.
- Code-switching. The localized corpus with code-switching normalized away, with English spans machine-translated into the target language. This directly tests CroCoSum's pretraining paradox in the code setting.
- Base model. A general monolingual model (IndoBART for Indonesian) fine-tuned on the same data. This asks whether code awareness or language nativeness carries the gain.
Why not just…?
Why not just generate in English and machine-translate? This is the real threat. Translation quality is high for Chinese and Indonesian, and most comments may be literal enough to survive. CroCoSum's finding that pipelines lose was on news text, not code, so it does not settle the question. The divergence-stratified comparison is designed to decide it. If the localized-minus-pipeline gap is flat across divergence quintiles and near zero, the conclusion is that translation suffices, and that is a useful negative result for practitioners.
Why not just fine-tune an existing multilingual code model on native comments? This may capture most of the gain if the hard part is supervision rather than representation. The ablation ladder answers it directly: multilingual fine-tuning alone, then adding the tokenizer, then adding target-language continued pre-training. The SeaLLMs compression result predicts that tokenization will matter most for Chinese. We report each step's increment separately.
Why not just prompt a large LLM? DocChecker's 124M model beat CodeLlama-34B by about 10% on an English documentation task, and CroCoSum's fine-tuned mBART-50 beat zero-shot GPT-3.5. Still, neither result is about non-English code comments, so we run this baseline rather than assume its outcome. Inference cost per function is reported alongside quality.
Why not just copy identifiers verbatim and translate the rest? Identifier-copying addresses part of CroCoSum's erroneous-switch errors cheaply. We add it as a post-processing variant of the pipeline. If it closes the gap, the advantage was about identifiers rather than about language. The CroCoSum error taxonomy, applied to every system, will show which errors remain.
Proposed Experiments and Decisive Checks
All systems are evaluated on both language test sets. Automatic metrics are chrF and BERTScore. Because the CodeSearchNet Challenge showed that docstring-overlap metrics can mislead, we add two other measures. The first is a functional proxy: how well the generated comment retrieves its own function among 1,000 distractors, measured by mean reciprocal rank with a fixed multilingual retriever. The second is native-developer ratings of adequacy and naturalness, together with the CroCoSum over-switch and erroneous-switch rates.
The decisive check is pre-registered. We will regress the per-item human-rated advantage of the localized model over the pipeline on the divergence score $d(c)$, controlling for comment length and engine disagreement. The hypothesis is supported only if the slope is positive and the gap in the lowest quintile is small.
Two secondary checks follow:
- Code-switching. The normalized-code-switching ablation should raise erroneous-switch rates, extending CroCoSum's paradox to code.
- Forgetting. English CodeXGLUE summarization should not degrade beyond a pre-set tolerance.
A failure of the decisive check, either a flat gap or a gap explained by engine disagreement, would falsify the claim that divergence predicts where localization matters. Even in that case the benchmark and the ablation ladder stand as contributions.
References
- Detecting Non-compositional MWE Components using Wiktionary
- Helsinki-NLP at SemEval-2022 Task 2: A Feature-Based Approach to Multilingual Idiomaticity Detection
- BERT-based Idiom Identification using Language Translation and Word Cohesion
- IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
- SeaLLMs -- Large Language Models for Southeast Asia
- NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
- CroCoSum: A Benchmark Dataset for Cross-Lingual Code-Switched Summarization
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- Bootstrapping Code-Text Pretrained Language Model to Detect Inconsistency Between Code and Comment
- The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
- EditSum: A Retrieve-and-Edit Framework for Source Code Summarization
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search