Term-Aware Activation Budgets for Multilingual Domain Fine-Tuning

Fine-tuning a translation or language model on a small specialized corpus, such as clinical notes, course catalogs or legal text, is often limited by GPU memory. The limit comes mostly from the intermediate results ("activations") that backpropagation must keep from the forward pass, not from the weights. Methods that save memory by storing activations for only some tokens, or storing them in compressed form, decide what to keep with signals that do not know which language or domain the token belongs to. We argue this is the wrong choice for multilingual domain adaptation.

The tokens that matter most in this setting are the pieces of domain terms, and multi-word terms in particular. Non-Latin scripts and rare technical words get split into many small subword fragments, so each fragment looks individually unimportant. We propose to allocate activation memory per token using two signals: how well that subword is shared with, or alignable to, a high-resource language, and whether it belongs to a domain term. Well-shared anchor tokens get cheap low-rank storage. Poorly shared term fragments keep exact activations. The claim is that at the same memory budget, this allocation preserves terminology accuracy that attention- and gradient-based token dropping loses.

Core Claim and Where Each Direction Sits

The core mechanism comes from cross-lingual subword sharing. Shared strings act as "semantic anchors", tokens whose embeddings are trained by high-resource data and stabilize the representations around them (Conneau and Lample; Patil et al.). Across scripts, literal overlap stops predicting transfer. Statistical alignability does predict it: Beyond Literal Token Overlap scores subword pairs with the eflomal word aligner and reports a Spearman correlation of −0.64 with part-of-speech transfer F1 for different-script pairs, against −0.45 for literal distribution overlap. That paper only diagnoses. We turn its score into a training-time decision about which activations to keep.

In-domain fine-tuning is the application regime. It also supplies the failure we target. MAGMATic shows that domain adaptation raises the Term Hit Rate (the share of reference terms translated correctly) by 7.71 points. Even so, accuracy on multi-word terms stays at 53.65% against over 76% for single-word terms, a gap that corpus BLEU hides.

The activation memory bottleneck is the constraint. Activations account for 70–90% of peak fine-tuning memory (Memory-Efficient Fine-Tuning of Transformers via Token Selection; TokenSeek), and parameter-efficient methods such as LoRA do nothing about them. MAGNET documents a 3× token-count "tokenization tax" on Indic scripts relative to its adaptive tokenizer. That tax is also a 3× activation tax.

The quantum data loading direction enters as a methodological lens, not as hardware, and we say so openly. Its lesson is that the cost of getting data into a compact representation can erase the savings of that representation. It also supplies one concrete tool. CL-QAS loads classical vectors through tensor-train encoding, which writes a large tensor as a chain of small low-rank cores so that cost grows with rank rather than with full dimension. It proves that a higher tensor-train rank buys fidelity at the price of gradient variance, and that more data tolerates lower rank. We reuse the tensor-train format for stored activations and carry over the rank–variance intuition: tokens seen often across languages can be stored at lower rank. The quantum fine-tuning papers (Quantum-PEFT; QTHA; Quantum LLM Fine-Tuning; QPEN) compress weights, classification heads, or cross-lingual features. None of them touches activation memory or term accuracy. That is the gap we occupy, and we do not claim a quantum advantage.

Research Program

We first compute, once per fine-tuning corpus, a per-subword alignability score $a_t \in [0,1]$ by running eflomal on the in-domain parallel data. This data is small (the extreme-efficiency results of Duh et al. involve about 1,000 examples), which removes the preprocessing-overhead objection raised against alignability. We also compute a term-membership flag $m_t$ from the terminology annotation where one exists, or otherwise from a term-extraction pass.

During training, each token $t$ at each layer receives a storage rank $r_t$ for its input activation. Full rank means exact storage. A small rank means tensor-train or truncated-SVD storage in the style of INSTANT and HOSVD activation compression. Rank zero means the token is dropped from the backward pass, as in TokenSeek. The ranks are chosen to satisfy a per-layer memory budget $B$:

$$\sum_{t=1}^{S} r_t\,(d + c) \le B, \qquad r_t \propto \big(m_t + (1-a_t)\big)\, g_t,$$

Here $S$ is sequence length, $d$ is hidden width, $c$ is the small per-rank overhead of the stored basis, and $g_t$ is TokenSeek's gradient-saliency signal, reused unchanged. The allocation therefore reduces to TokenSeek when $m_t$ and $a_t$ carry no information. Each ingredient can be switched off in ablations. We will analyze whether the gradient-variance growth predicted for low-rank loading in CL-QAS appears as noisier updates on low-rank tokens. That analysis tells us whether the rank floor for anchors must scale with how often they appear in the corpus.

Baselines

The naive reference is full-activation in-domain fine-tuning with no memory saving, following the recipe of Optum's WMT22 biomedical submission. That submission reports 38.12 BLEU on the WMT biomedical test set after pure in-domain fine-tuning of opus-mt-en-es (36.89 before), with general-domain BLEU falling from 41.0 to 34.74. We rerun this recipe as the memory-unconstrained ceiling. We also report general-domain retention, because a memory method that implicitly regularizes toward less forgetting would otherwise look better than it is.

The strongest existing method on the exact question is TokenSeek. It keeps activations only for tokens ranked highly by cumulative attention and penultimate-layer gradient. It reports Llama3.2 1B QLoRA memory of 2.8 GB (14.8% of the baseline) and an Open-Platypus five-benchmark average of 52.61 against 40.82 for full-token training. It has never been tested multilingually, and its token-level decision is the thing we modify. We run it on identical data at an identical peak-memory budget, with its weighting factors tuned on the same validation budget we give our method.

The strongest uniform-compression competitor is INSTANT. It projects all activations into a periodically recalibrated low-rank subspace and reports up to 32× activation-memory reduction with under 1% accuracy loss. We also include HOSVD compression using the released ActivationCompression code, which reports 18.87× lower memory than matrix SVD on vision tasks. Both methods are matched to our peak memory.

Exact activation checkpointing via Rockmate is compared at matched wall-clock time rather than matched memory, since it trades compute for memory with no loss in gradient quality.

The key ablations hold the budget fixed and change only the allocation: random rank allocation, term flag alone, alignability alone, and literal-overlap scores in place of eflomal alignability. The last isolates whether alignability matters beyond string identity. We count our method as beating TokenSeek if it improves multi-word Term Hit Rate at equal memory, by a margin exceeding seed variance over at least three seeds, without losing more than 0.5 BLEU.

Why not just…?

Why not just use TokenSeek as is? Nothing in the file shows that TokenSeek fails on domain terms, so this is a real threat. Our first experiment is a diagnostic: on unmodified TokenSeek, we measure the retention rate of term-fragment tokens against other tokens. If TokenSeek already keeps term fragments at or above the base rate, the central premise is false and we will report that rather than proceed. Our reason to expect failure is structural. Attention mass and gradient magnitude are split across the many pieces of an over-segmented term, and MAGMATic's 53.65% multi-word accuracy shows these units are already fragile without any dropping.

Why not just compress everything uniformly with INSTANT or HOSVD? Uniform low rank is simpler and achieves large reported ratios. However, those results come from classification, pre-training and vision, where no small token subset carries the evaluation signal. The matched-memory comparison settles the question. If uniform compression matches our multi-word Term Hit Rate, per-token allocation is unnecessary.

Why not just shorten sequences with a better tokenizer or transliteration? MAGNET's 3× compression and transliteration (Happiness is Sharing a Vocabulary) cut activation memory at its source, and they complement our method. They require retokenizing or retraining the base model, which most practitioners fine-tuning a released model cannot do. Where MAGNET-style models exist, we will stack our allocation on top of them and test whether any gain survives.

Why not just checkpoint activations? Checkpointing gives exact gradients for roughly one extra forward pass. It will likely win whenever a single sequence fits in memory. Our method only matters when the budget forces batch or sequence truncation, and the matched-wall-clock comparison will show where that crossover lies.

Proposed Experiments and Decisive Checks

We use three settings. The first is English–Spanish clinical translation on WMT22 ClinSpEn with opus-mt-en-es, evaluated by BLEU on the test set and on EMEA/M&R. The second is Italian–English academic translation fine-tuned with MAGMATic-style terminology and evaluated by Term Hit Rate, reported separately for single- and multi-word terms. The third tests Llama3.2 1B with QLoRA on the same two tasks, which lets us compare directly against TokenSeek's reported setting.

Both domain benchmarks share a script, so we add a different-script stress test. We fine-tune on in-domain-style FLORES-200 subsets for Hindi and Urdu, where literal overlap is near zero but alignability is not, and measure retention with FLORES-200 and zero-shot XNLI. No reported result was found on Lacuna for memory-constrained domain fine-tuning in any different-script pair, so this setting has no external reference number.

Memory is reported as peak GPU memory and time as wall-clock time per step, including the eflomal preprocessing cost. Because learned metrics miscalibrate in unseen domains (Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains), we treat BLEU and Term Hit Rate as primary and learned metrics as secondary.

Three checks decide the outcome. The first is the TokenSeek retention diagnostic. The second is the literal-overlap-versus-alignability ablation on the Hindi–Urdu pair, where alignability should matter most if it matters at all. The third is the matched-memory multi-word Term Hit Rate comparison against both TokenSeek and uniform compression.