When a language model is adapted to medicine in a language with little medical text, such as Japanese, Vietnamese, or Swedish, practitioners pad the training mix with English medical text. The current recipe is static. A fixed ratio of English, target-language, and machine-translated documents is chosen in advance and streamed for the whole run. Two findings in the literature pull against each other here. More English medical text is the strongest single predictor of Japanese medical performance, yet indiscriminately adding tokens or domains also causes "drowning" and negative interference. We propose to resolve this by letting the model being trained decide which English text it reads next.
The idea borrows the core move of a flipped classroom: the learner attempts the material first, and the expert's explanation is aimed at what the learner got wrong. At regular intervals during continued pre-training (further pre-training an existing model on new text), we probe the model on medical concepts in the target language. We measure whether its internal representation of each concept matches its representation of the same concept in English. English documents are then sampled in proportion to how badly each concept is aligned. English text goes where the target language is weak, and is withheld where it would only crowd out target-language nuance. If this works, it yields both a better adaptation recipe and a measurable account of when English acts as scaffolding and when it acts as noise.
Core Claim
At a fixed token budget, a curriculum that picks English medical documents by per-concept cross-lingual alignment gaps will beat the best static English–target mixture on target-language medical benchmarks. It will also beat a curriculum that picks documents by the model's loss alone. The cross-lingual signal, not adaptivity in general, is the claimed ingredient. A secondary claim is diagnostic. Each concept's initial alignment gap should predict how much that concept gains from added English text, and concepts whose gap is already small should gain little or lose. This would turn the open contradiction in the file (Gap 2) into a testable relation.
Grounding in Prior Work
Several papers motivate the gap. Leveraging High-Resource English Corpora for Cross-lingual Domain Adaptation in Low-Resource Japanese Medicine via Continued Pre-training continues pre-training a 13B Japanese–English model on six fixed corpus recipes. It finds that English token count correlates most strongly with Japanese medical scores, and that the largest recipe (EnJa-Hybrid) loses to the balanced one. That paper therefore shows that both the amount and the choice of English matter, but it only ever searches over global ratios. Pointing the other way, Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-Resource Language Translation finds that auxiliary pre-training data hurts when it diverges from the target domain. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages finds a "Goldilocks" point beyond which extra related languages degrade results. Both are evidence that more data is not monotonically good, which is what our selector exploits at the level of individual concepts rather than whole corpora.
The probe comes from Can you map it to English? The Role of Cross-Lingual Alignment in the Multilingual Performance of LLMs. That paper shows with activation patching that middle-layer alignment between a non-English input and its English counterpart causally drives whether an instance transfers, and it gives instance-level alignment scores (DALI). It uses alignment only as an after-the-fact diagnostic. We use it as a training-time control signal.
The pedagogical framing comes from two sources. Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discovery inverts teacher–student training in vision, so that a student's discoveries reshape the teacher's attention keys. We take the inversion, but not the attention-editing mechanism, and we apply it to data selection in language pre-training. Its documented weakness, forgetting the old classes while chasing new ones, becomes a constraint we monitor through English-task retention. The human-curriculum work, including From Knowledge Transfer to Capability Development: The Future of PBL + CBL Teaching Method in Operating Room Nursing Education on the "illusion of competence," supplies the analogue we test. A model can be fluent in target-language medical text without having tied its terms to the conceptual knowledge it holds in English, and passive exposure does not reveal this.
En-ViMedNER and Multilingual Clinical NER: Translation or Cross-lingual Transfer? constrain the design. Both find that translation pipelines corrupt entity boundaries, and En-ViMedNER finds that translation BLEU does not track entity accuracy. We therefore do not rely on translated text to define concept pairs where a dictionary-level pivot is available.
Research Program
The first component is a concept index. English PubMed Central text and target-language text (J-STAGE for Japanese, Vietnamese PubMed-derived text for Vietnamese) are tagged with medical concept identifiers. Where target-language entries exist, we use a shared ontology, the Unified Medical Language System (UMLS) used by En-ViMedNER. Where coverage is thin, we fall back to term pairs mined from aligned abstracts. Each document is then a bag of concept IDs. The index serves two purposes: a pool of English documents per concept, and a set of matched English–target mention pairs per concept for probing.
The second component is the attempt phase. Every $T$ training steps, the current model encodes a held-out sample of matched mention pairs. For each concept $c$ we compute a middle-layer alignment score $a_t(c)\in[0,1]$, adapting DALI, together with an English competence check: cloze accuracy on masked English mentions, $e_t(c)$. The gap $g_t(c)=e_t(c)\,(1-a_t(c))$ is large only when the model knows the concept in English but has not connected it to the target language. That is the case in which English text should help. A concept the model does not know in English either produces a small gap, because English text is not expected to fix that through transfer.
The third component is the lecture phase. All scarce target-language text is streamed at a fixed rate in every condition. The selector controls only which English tokens fill the English share, and how many. Concept $c$ is sampled with probability
$$p_t(c)\propto n_c^{\alpha}\,\exp\big(\beta\,g_t(c)\big),$$
where $n_c$ is the number of English documents for concept $c$, $\alpha$ tempers raw frequency, and $\beta$ sets how strongly the gap steers sampling. A document is then drawn from that concept's pool. A variant also lets the total English share shrink once the mean gap falls below a threshold, which tests whether the "right" English ratio changes over the course of training.
The fourth component is analysis. We log per-concept gap trajectories and relate initial gap to final per-concept gain on downstream items tagged with that concept. We also check whether concepts receiving little English text lose accuracy relative to the static recipe, as a drowning account would predict for concepts receiving heavy English text.
Baselines
The naive reference point is the base bilingual model with no medical continued pre-training. A second naive point is continued pre-training on target-language medical text alone (the JaDominant-style recipe). This shows how much any English contributes.
The strongest existing method is the EnJa-Balance static recipe from Leveraging High-Resource English Corpora for Cross-lingual Domain Adaptation in Low-Resource Japanese Medicine via Continued Pre-training. That paper reports an average score of 0.475 on the Japanese JMedBench tasks for a 13B LLM-jp model, against 0.468 for EnDominant and 0.453 for JaDominant. That figure is for their scale and data. Because our compute will likely require a smaller model, we will rerun EnJa-Balance ourselves with the same base model, the same target-language text, the same total token budget, and the same English share as our method. We will also give it a small sweep over the English ratio with the same tuning budget we spend on $\alpha$ and $\beta$. Beating it means a higher average on the 20 Japanese JMedBench tasks across three seeds, by more than the seed spread, with no drop beyond seed noise on the 7 English tasks.
Two ablations isolate the key ingredient. The first is loss-driven selection: English concepts are sampled by the model's own cloze loss, with no cross-lingual term. It is adaptive, but it is not flipped across languages. The second is topic-matched static selection: English documents are chosen once, before training, to match the concept distribution of target-language text. It is targeted, but it ignores the learner. If either matches our method, the claim that learner-measured cross-lingual alignment matters is wrong. A third control replaces the alignment score with random scores of the same variance, to rule out a benefit from churn alone.
For Vietnamese, the in-language reference on En-ViMedNER is ViPubMedDeBERTa-base fine-tuned on Vietnamese, which reports an F1 of 52.70. The best few-shot language model reported there is Gemini 3.1 Flash at 36.47 F1. We will fine-tune our adapted model and the static-recipe model identically for NER and compare both against these references.
Why not just…?
Why not just tune the static mixing ratio harder? The Japanese study already searched six recipes, and the spread among the balanced ones is under one point, from 0.468 to 0.475. This suggests a ratio search is near its ceiling. Our hypothesis is that the gains lie in which English documents are used, which no global ratio can express. The ratio-swept EnJa-Balance rerun is exactly this alternative given equal tuning effort. If it ties our method, the proposal fails.
Why not just select data by the model's loss, as generic active curricula do? This is the most serious threat, because high-loss English documents may already concentrate on the concepts the model handles poorly. The difference is that loss measures English difficulty, while our gap measures disconnection between languages. Spending budget on concepts the model does not know in English may teach English medicine without helping Japanese. The loss-driven ablation settles this. We additionally check whether its selected concepts overlap with high-gap concepts. Heavy overlap would mean the cross-lingual probe is an expensive proxy, and we would report it as such.
Why not add an explicit loss that pulls middle-layer English and target representations together, instead of changing the data? That is a direct route, and the alignment paper's causal result makes it plausible. But a forced alignment loss on concept pairs risks collapsing target-language distinctions, which is the cultural and clinical misalignment risk the file raises. It also depends on pair quality, which the NER papers show is fragile. We run it as a comparison arm with the same pairs and a tuned weight. If it wins, the conclusion is that the probe is valuable as a loss rather than as a sampler, which is still informative.
Why not just translate the English corpus or use translate-test? Machine-translated text was already part of the Japanese recipes and did not beat original English. The two NER studies find that translation corrupts entity boundaries. We therefore treat translation as already covered by the strongest baseline rather than adding a separate arm.
Proposed Experiments
The main experiment is Japanese continued pre-training on J-STAGE plus a selected subset of PubMed Central. It compares the no-CPT, Japanese-only, rerun EnJa-Balance, loss-driven, topic-matched, random-score, alignment-loss, and learner-led conditions at one fixed token budget and three seeds. Evaluation uses all 27 JMedBench tasks (JMedBench), reported separately for Japanese and English. JMedBench reports no numeric scores in its summary, so the rerun baselines are our reference.
Transfer to a second language uses Vietnamese with En-ViMedNER, where UMLS annotations make the concept index reliable. It is scored by NER F1 for monolingual Vietnamese and by Judge-F1 for the English-input setting, where NLLB-M1 reports 45.44.
A third check uses Swedish, a language with little medical text and only an evaluation set. We adapt with Swedish medical text where it is obtainable and report MedQA-SWE accuracy (MedQA-SWE). GPT-4 at 84.3% and random choice at 20.0% serve only as bounds. We do not expect a small open model to approach GPT-4. The comparison of interest is against the static recipe at equal budget.
Three decisive checks determine whether the claims hold. First, the learner-led condition must beat both the ratio-swept static recipe and loss-driven selection on the Japanese average. Second, a concept's initial gap must positively predict its per-concept gain, with near-zero or negative gain for concepts that start well aligned; this is the explanation of Gap 2. Third, English-task scores must not fall more than the static recipe's, which guards against the forgetting failure FlipClass exhibits.
Failure on the first check together with success on the second would still be a useful negative result. It would mean the alignment gap explains when English helps, but that steering sampling is not the way to act on it.
References
- Leveraging High-Resource English Corpora for Cross-lingual Domain Adaptation in Low-Resource Japanese Medicine via Continued Pre-training
- Exploiting Domain-Specific Parallel Data on Multilingual Language Models for Low-Resource Language Translation
- Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages
- Can you map it to {E}nglish? The Role of Cross-Lingual Alignment in the Multilingual Performance of {LLM}s
- Flipped Classroom: Aligning Teacher Attention with Student in Generalized Category Discovery
- From Knowledge Transfer to Capability Development: The Future of PBL + CBL Teaching Method in Operating Room Nursing Education
- En-ViMedNER: An English-Vietnamese Parallel Biomedical Corpus with UMLS Semantic Type Annotations
- Multilingual Clinical NER: Translation or Cross-lingual Transfer?
- JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models
- MedQA-SWE - a Clinical Question & Answer Dataset for Swedish