Where Cross-Lingual Contract-to-Code Breaks: A State-Machine Pivot Test

Language models can now turn a written requirement ("the buyer may withdraw the deposit if delivery is not confirmed within 30 days") into a smart contract, a small program on a blockchain that moves money automatically when conditions are met. Every benchmark for this task is in English. Real agreements are drafted in Chinese, German, Arabic, and Hindi. The obvious next step is to run the English benchmarks in other languages and report the drop. This proposal argues that the drop matters less than where it happens. In contract code, a translation slip that turns "may" into "must", or moves the scope of "unless", produces code that compiles, passes most tests, and pays the wrong party. We propose to measure cross-lingual transfer one meaning-bearing slot at a time. We do this by sending generation through an explicit state machine, a list of contract states and the guarded transitions between them. That gives a place to check, in any language, whether obligations, permissions, conditions, parties, amounts, and deadlines survived.

The concrete bet has three parts. First, a requirement-to-state-machine extractor trained only on English, attached to a multilingual model through small adapter layers, transfers to other languages better than end-to-end code generation does. Second, its residual errors concentrate on modal force and condition scope, while numbers and dates mostly survive. Third, which slots will fail in which language can be predicted cheaply, before any code is generated, from how well the trigger words line up across languages in aligned static word vectors. If machine translation into English followed by an English pipeline matches us on these slots, we will say so, and the benchmark and diagnostic become the contribution.

Core Claim

Zero-shot cross-lingual transfer means training on English task data and testing on other languages with no target-language examples. For contract-to-code, it fails selectively, not uniformly. The closest prior work is entirely English. SolEval benchmarks repository-level Solidity generation from human-written English descriptions and names non-English requirements as future work. FSM-SCG shows that generating an intermediate state machine (SmartFSM) before code raises compilation and lowers vulnerability risk, but its requirements and state machines are English, written with GPT-4. Legal Compliance Evaluation of LLM-generated contracts compares generated code to hand-built Petri nets of English agreements. None of these asks whether the state machine can act as the language boundary. In our design, the only multilingual step is text-to-state-machine, and everything downstream is language-free and checkable.

Two lines of cross-lingual evidence motivate the slot-level hypothesis. MLQA found that temporal questions transfer most easily, because dates and numbers act as fixed anchors, and that distant languages lose much more (XLM: Spanish 68.0 F1 versus Hindi 48.8 F1). MultiEURLEX constrains the design. In legal classification, full fine-tuning on English erodes cross-lingual alignment, and adapters that update about 0.05% of weights recover up to 6.8 points of mean R-Precision. SolEval's own finding that fine-tuning is essential (Qwen-7B Pass@5 from 16.67% to 58.83%) therefore collides directly with transfer, and we have to test this explicitly.

Word embeddings enter as a diagnostic lens, not as the generator. Specializing Word Embeddings for Similarity or Relatedness shows that ordinary distributional vectors place related words close even when they are not similar. "May", "must", and "must not" are exactly such words. The same property makes soft-matching metrics like ROUGE-WE, which gives partial credit for near-synonyms, unsafe for scoring contracts. We use it the other way round. If a target-language modal or connective's nearest aligned English neighbor is the wrong modal, or a diffuse cluster of modals, we predict that slot will fail.

Research Program

The first piece is data. We take roughly 300 SolEval tasks whose human-annotated descriptions are self-contained, and pair them with SmartFSM-style state machines. We start from FSM-SCG's format and have them checked by two Solidity developers. We then have the descriptions professionally translated into six languages chosen for typological spread and overlap with MLQA: German, Spanish, Vietnamese, Chinese, Arabic, and Hindi. Translated test sets carry translationese that flatters machine translation. To avoid this, bilingual developers will also natively draft about 50 requirements each in Chinese, German, and Hindi from the reference code alone.

Each item also gets minimal pairs: copies where exactly one slot is changed. The changes swap obligation for permission, add or remove a negation, move the scope of an exception, swap the actor, or change an amount or deadline. Each pair comes with a unit test that tells the two versions apart. For a small legal-agreement check, we translate the two simplest agreements from the Petri-net study, whose reference nets already exist, into Chinese and German.

The second piece is the pivot system. A multilingual open model of the size FSM-SCG used (7–8B) gets adapters trained only on English requirement-to-SmartFSM pairs from the FSM-SCG data. A separate English-only generator maps the state machine to Solidity, using FSM-SCG's compile-and-Slither repair loop. Because the generator never sees the target language, any cross-lingual loss must enter at extraction, where we can score it slot by slot against the gold state machine.

The third piece is the alignment predictor. For each slot type, we find the trigger words in each target-language requirement: modals, negators, conditional and exception connectives, and role nouns. We map them into English through cross-lingual aligned static vectors, such as subword vectors mapped into a shared space with a bilingual dictionary. Each trigger gets a score: the cosine margin between its correct English counterpart and the nearest competing member of the same slot class. Low margin should predict extraction error. We also test whether sending only low-margin slots to a second check, either re-extraction or a targeted question to the model, recovers most of the gain of checking everything.

Baselines

The do-nothing reference is the English pipeline fed the target-language text directly. This is the SolEval-style fine-tuned generator with no pivot and no adapters, and its gap to the English score defines the transfer gap we are trying to close. On English, the strongest reported numbers are 26.29% Pass@10 for DeepSeek-V3 without task fine-tuning and 58.83% Pass@5 for SFT Qwen-7B, both from SolEval. The file also quotes 58.33% for the latter, so we will re-measure it. For FSM-SCG, the reported figures are 95.1% compilation on LLaMA3.1-8B and 53.21% zero-risk contracts on Qwen2.5-7B (FSM-SCG). We found no reported result on Lacuna for any non-English smart-contract generation setting, so every target-language number is new.

The strongest practical competitor is translate-test. A strong commercial or open translation system maps the requirement to English, and the best English pipeline (SFT generator, and separately FSM-SCG) runs on the output. It is strong because modern translation of short technical text is good and it reuses all English supervision. It gets the same English training data, the same generator, and the same repair-loop budget. The second strong competitor is direct prompting of a frontier multilingual model with the target-language requirement and a plan-then-code prompt, as in the Petri-net study. It gets the same number of samples per task and the same compile-and-Slither feedback rounds. The third is FSM-SCG run with full-parameter fine-tuning on the multilingual base.

Two ablations isolate the ingredients. Pivot-with-adapters against pivot-with-full-fine-tuning isolates forgetting. Embedding-triggered selective checking against an equal-size random selection of slots, and against checking all slots, isolates the predictor.

Beating a baseline has a specific meaning here. We count a win when the pivot system has higher slot accuracy on modal and scope slots, and higher minimal-pair discrimination, averaged over the six languages with a paired bootstrap over tasks. It must do this without a Pass@k loss of more than two points against the same baseline. Aggregate Pass@k alone does not count, because it barely registers a flipped permission.

Why not just…?

Why not just translate the requirement into English and use the English pipeline? This is the real threat, and the file's evidence against it is old. Hsu et al. found that training on machine-translated data underperformed zero-shot mBERT transfer for reading comprehension, and MLQA reported multilingual models beating translation baselines. Both predate strong modern translation. Our distinct prediction is narrower. Translation should preserve numbers and parties but sometimes neutralize modal force and exception scope, for example where Chinese or Hindi mark obligation softly. The minimal-pair test settles this directly. If translate-test discriminates modal and scope pairs as well as the pivot does, we will report that translation is sufficient for this task, and we will test whether combining it with the state-machine check still helps.

Why not just prompt a frontier multilingual model directly? Frontier models read these languages well. But the Petri-net study found that even in English they trade coverage for invented behaviors, or drop edge cases, once contracts grow beyond a few steps. In direct prompting, cross-lingual misreadings and coding errors are tangled together in the output. We run this baseline at matched sampling budget, and we also score its intermediate plan with the slot metrics where the plan states a state machine.

Why not just fine-tune everything on multilingual contract data? No parallel corpus of non-English requirements paired with Solidity exists at scale. MultiEURLEX shows that full English fine-tuning damages transfer in the legal domain. The adapter-versus-full ablation tests whether that finding carries over to generation.

Why not drop the embeddings and let the model flag its own uncertainty? That is a fair alternative. We include model self-reported confidence and token probability as competing predictors of slot error. The static-vector score earns a place only if it predicts errors at least as well, at a fraction of the cost, before any generation.

Proposed Experiments

The decisive check is a single plot per language of slot accuracy by slot type for pivot, translate-test, direct prompting, and the do-nothing reference. The core claim predicts that the drop from English is largest on modal and scope slots and smallest on amounts and deadlines. It also predicts that the pivot narrows the modal and scope gap most for Arabic and Hindi. If errors turn out uniform across slot types, the slot-level framing is wrong, and we fall back to reporting the benchmark.

Second, minimal-pair discrimination: the fraction of pairs whose generated contracts behave differently under the distinguishing unit test. We report it for all systems, on translated and natively drafted subsets separately, to expose translationese effects.

Third, for the predictor, we report the area under the ROC curve of the alignment margin for predicting slot error across all languages and slot types. We also report the recall-versus-budget curve for selective checking against random and full checking. We count success if checking the lowest-margin 20% of slots recovers at least half of the full-check gain.

Fourth, we report SolEval Pass@k, Compile@k, and Vul@k, plus FSM-SCG's compile and zero-risk rates, per language. These confirm that the slot-level gains do not cost functional correctness. Finally, functional-equivalence scores on the two translated Petri-net agreements show whether the slot-level results survive end to end on real agreements.