Thinking in a Fluent Language: Surprisal-Gated Pivot Reasoning for Low-Resource Code Generation

Open code models write Python well and most other languages worse. One known remedy is to let the model solve the problem in a language it knows well and then translate the result. Handing a model a Python solution as a hint ("translation mode") sharply raises accuracy in the target language (Multi-lingual Evaluation of Code Generation Models). Always taking this detour has costs, though. It doubles generation length, it introduces translation errors, and Python itself turns out to be a poor transfer partner for many languages (Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study). We propose that the model's own surprisal should make the choice, where surprisal is the negative log-probability a model assigns to each token it reads. Before committing to an answer, the model drafts the opening lines of a solution in the target language and in a few candidate "pivot" languages, then measures how surprised it is by each draft. It reasons in the pivot only when the target-language draft is unusually surprising for that language, and it picks the pivot on which it is least surprised.

The core idea fits in two sentences. A model's surprisal on its own draft code measures how "fluent" it is in that language on that problem, much as surprisal measures reading effort for humans. Using this signal per problem to decide whether to take a detour, and through which language, should match or beat always translating from Python at a fraction of the tokens, and it should transfer across languages without re-tuning a threshold for each one.

Core Claim

We make three claims, and each can be falsified. The first is that raw uncertainty thresholds do not transfer across programming languages, because baseline predictability differs by language: verbose, boilerplate-heavy Java is far more predictable token by token than terse Ruby. Once uncertainty is normalized against the model's own surprisal distribution on ordinary code in that language, a single threshold tuned on Python should work for C++, Go, Kotlin, Ruby and others. The second is that the difference in normalized surprisal between the target draft and a pivot draft predicts, per problem, whether pivoting will help. The third is that choosing the pivot per problem beats a fixed Python pivot. The gain should be largest for targets where the transfer study found Python to be a poor partner.

We write $s_\ell(x)$ for the mean first-token surprisal over the first $k$ lines of a greedy draft for problem $x$ in language $\ell$. First-token surprisal is $-\log P$ of the first token of each line. We write $F_\ell$ for the empirical distribution function of that same quantity on a reference sample of unlabeled, correct code in $\ell$. The normalized score is $u_\ell(x) = F_\ell(s_\ell(x))$, which is the percentile of the draft's surprisal within its own language. The decision rule is

$$\ell^*(x) = \begin{cases} t & \text{if } u_t(x) < \tau \\ \arg\min_{\ell \in C} u_\ell(x) & \text{otherwise,} \end{cases}$$

where $t$ is the target language, $C$ is a small candidate pivot set, and $\tau$ is a single percentile threshold shared across all languages. When $\ell^* \neq t$, the model writes a short plan and a full solution in $\ell^*$, then translates that solution into $t$.

How this sits relative to prior work

Two papers supply the mechanism, and both stop at Python. Uncertainty-Guided Chain-of-Thought for Code Generation with LLMs measures entropy or the top-1/top-2 probability margin at the start of each code line. It triggers step-by-step reasoning only when uncertainty crosses a hand-tuned threshold, and this avoids "overthinking" on easy problems. We reuse its line-start gating. We differ in two ways: we normalize the threshold per language, and our trigger switches languages rather than inserting English reasoning in place. Pruning the Unsurprising shows that first-token surprisal of reasoning steps marks the steps that matter, and uses it to compress long reasoning traces for fine-tuning. We borrow first-token surprisal as the statistic, but apply it to code drafts at inference time and compare it across languages. That paper evaluates only Python and never asks whether the statistic's scale is comparable across languages.

Three papers motivate the pivot. Multi-lingual Evaluation of Code Generation Models built the transpiled multilingual benchmarks we use and found that a Python hint strongly lifts target-language generation. It always supplies the hint and never studies reasoning traces. Cross-lingual Transfer in Programming Languages finds that Kotlin and JavaScript are good donor languages and Python and C++ poor ones. It predicts transfer with a ranking model over language-pair similarity, but those predictions are fixed per language pair and are used for fine-tuning. Our choice is made per problem at inference time, from the model's own probabilities. CodeGeeX shows that translation quality is asymmetric between directions and names cross-language reasoning as open work. That asymmetry is exactly why the pivot direction must be chosen rather than assumed.

Two papers constrain the design. CodeCoT documents that narrative reasoning can produce "right but unrunnable" code, with a 27% syntax error rate on HumanEval before repair. That warns us that a translation step can introduce the same failure, especially in strictly typed targets, so we measure compile errors separately. Which Reasoning Trajectories Teach Students to Reason Better? shows that raw likelihood is a weak selection signal compared with a rank-adjusted one. We therefore test a rank-based variant of $u$ as an ablation rather than assuming mean surprisal is best.

The cognitive-modeling literature supplies the reason normalization is needed. Surprisal's effect on human reading time is itself modulated by overall text difficulty (Text difficulty modulates the surprisal effect in self-paced reading). Information density in real discourse also wiggles systematically with structure rather than staying flat (Tsipidi et al.). If the same number means different "effort" in different registers of human text, it should not be read on one scale across programming languages either.

Open weights are a requirement here, not a preference. The method needs token-level probabilities for every draft, which open models such as DeepSeek-Coder, Qwen-Coder and CodeLlama expose and which closed APIs typically do not. The parity question therefore becomes whether a mid-size open model with this gate closes part of its gap to its own Python performance.

Research Program

The first stage is a measurement study. For each model and each of roughly a dozen languages, we compute $F_\ell$ from a few thousand correct solutions, drawn from held-out CodeNet submissions in that language. We then plot pass@1 against raw and normalized line-start surprisal. The question is whether raw surprisal–error curves diverge by language while normalized curves roughly coincide. This alone would be a new result: the first cross-language characterization of uncertainty signals that existing gating methods assume but never test outside Python.

The second stage builds the gate. We use greedy drafts of the first $k$ lines, with $k$ set to 3 by default and then swept, in the target plus a candidate pivot set $C$ = {Python, JavaScript, Kotlin, Java}. $C$ comes from the donor findings above. The only tuned parameter is $\tau$, fitted on Python problems alone. Pivot execution has two steps: a short plan followed by a complete pivot-language solution, then a translation prompt into the target. We deliberately add no execution loop. This keeps the gate's contribution separable from repair, and CodeCoT-style repair can be layered on afterwards.

The third stage is the analysis that makes the result interpretable. We compute "pivot regret," the accuracy gap between our chosen pivot and an oracle that knows which of the candidate routes (no pivot, or each pivot) succeeds. We also break errors down into logic failures, translation errors and compile or type errors, separately for statically and dynamically typed targets.

Baselines

The naive reference point is direct zero-shot generation in the target language with greedy decoding. This is the "do nothing" condition, and the naive-CoT literature shows it is sometimes hard to beat. For context on what open multilingual models achieve directly, CodeGeeX reports pass@100 of 60.92% on Python in HumanEval-X for CodeGeeX-13B. Our metric is pass@1 with modern open models, so this figure is context and not a head-to-head target.

The strongest existing method on the pivot side is always-translate. The model solves the problem fully in Python, then translates, following the translation mode of Multi-lingual Evaluation of Code Generation Models. That paper reports "nearly 100% pass@100" on MathQA in this mode when a correct Python solution is given. For the self-generated Python pivot we run, no reported pass@1 result was found on Lacuna. We also run an always-translate variant through the best fixed donor, Kotlin or JavaScript, which is the strongest static reading of Cross-lingual Transfer in Programming Languages.

The strongest existing method on the gating side is UnCert-CoT (Uncertainty-Guided Chain-of-Thought for Code Generation with LLMs). That paper reports a 6.1% PassRate improvement on MHPP in Python. For non-Python languages, no reported result was found on Lacuna. We run it in two ways: with its threshold tuned once on Python and applied everywhere, and with an oracle threshold tuned separately on each target language's test split. The oracle version is an upper bound for that method.

We also run a compute-matched sampling baseline. It draws as many target-language samples as our method spends in tokens and selects among them by mean log-probability. This asks whether we are simply buying accuracy with extra tokens.

Three ablations isolate the key ingredients. The first uses raw rather than normalized surprisal, which isolates normalization. The second sends the same fraction of problems to a pivot, but chooses them at random, which isolates the gate. The third always uses a fixed pivot when the gate fires, which isolates per-problem pivot choice.

All conditions use the same models, prompts, decoding temperature and benchmarks. Any tuning uses Python problems only, except the explicitly labeled oracle variants. We report tokens generated and wall-clock latency alongside accuracy. Beating a baseline has a specific meaning here. Against always-translate, we need equal or higher mean pass@1 across non-Python languages at substantially fewer tokens, plus a clear gain on the targets where Python is a poor donor. Against Python-tuned UnCert-CoT, we need higher pass@1 on non-Python targets. Coming within noise of the per-language oracle UnCert-CoT without per-language tuning also counts as success.

Why not just…?

Why not just always translate from Python? This is the real threat. If a self-generated Python solution is correct most of the time, always translating may match our accuracy, and our only advantage would be cost. The experiments settle it directly. If always-translate matches us on accuracy across all targets, we will report the method as a cost-saving gate and not claim an accuracy gain. Our specific accuracy prediction is confined to targets such as C++ and to strictly typed languages. The transfer study's finding that Python is a poor partner gives a concrete reason to expect a fixed Python pivot to underperform there.

Why not just use UnCert-CoT per language with its own threshold? That works if one is willing to label data in every language, which is exactly what is scarce for low-resource targets such as Ruby, Perl or Swift. The per-language oracle baseline measures how much is lost by not tuning. The raw-versus-normalized ablation tests whether normalization recovers that loss. UnCert-CoT also reasons in place, in the language it is already struggling with. Our pivot condition tests whether moving the reasoning elsewhere adds anything beyond gating.

Why not just sample more and rerank? Reranking by likelihood or pragmatic scoring, as in Pragmatic Reasoning improves LLM Code Generation, helps in Python. However, every sample it draws comes from the same weak target-language distribution. The compute-matched sampling baseline tests whether our gains are only extra compute. If it matches us, the pivot adds nothing, and we will say so.

Why not just fine-tune on the low-resource language? Fine-tuning from a related language does beat zero-shot (Cross-lingual Transfer in Programming Languages), and it is a complementary route rather than a rival. Our method needs no training data in the target language and applies to any open checkpoint. As a secondary check, we apply the gate on top of a fine-tuned checkpoint, to see whether the two effects add.

Proposed Experiments and Decisive Checks

The benchmarks are HumanEval-X (Python, C++, Java, JavaScript, Go) and the MBXP suite with Multilingual HumanEval (adding TypeScript, Ruby, Kotlin, PHP, C#, Scala, Swift, Perl), run through amazon-science/mxeval. MHPP serves as the Python anchor, where UnCert-CoT's reported gain lets us check our reimplementation. The models are open code checkpoints from the DeepSeek-Coder, Qwen-Coder and CodeLlama families at two sizes each. Metrics are pass@1, tokens per problem, latency, compile-error rate, and pivot regret.

Three checks would decide the question. The first concerns normalization. Fit $\tau$ on Python and apply it to every other language. If normalized gating does not beat raw gating on non-Python targets, the first claim fails. The second concerns the pivot signal. Across problems, the normalized surprisal gap $u_t(x) - \min_{\ell} u_\ell(x)$ should correlate positively with the realized benefit of pivoting, meaning pivot success minus direct success. We also require the gate to beat the random-routing ablation at equal pivot rate. A null correlation would mean surprisal does not track fluency in a usable way. The third concerns pivot choice. Per-problem choice must beat the fixed Python pivot on C++ and on at least one strictly typed target, while the compile-error breakdown shows that translation into strictly typed languages is not where the gains are lost. If all three checks pass, a single signal taken from a model's own probabilities tells it both when to take a detour and which language to take it through, with no labels in the target language.