Bilingual speakers often switch languages in the middle of a sentence. This is called code-switching. NLP models handle it poorly because almost no written training data contains it. The cheap fix is to take ordinary sentences and swap random words for their dictionary translations. Researchers call this syntactic-agnostic substitution, because it ignores grammar. It produces text that real speakers would never write. It also measurably damages what pretrained models encode about meaning: in Code-Mixed Probes, semantic-similarity correlation with English stays at 0.85–0.87 for natural Spanglish but falls to nearly zero for randomly substituted text. The more grammatical alternatives need resources that low-resource language pairs lack. These include phrase tables built from parallel corpora, as in Matrix Language Frame augmentation, and contextual word alignments, as in Borisov et al..
We borrow one idea from robotics. There, the kinetic-energy metric measures the cost of a joint motion as $\dot q^T M(q)\dot q$, where $M(q)$ is the robot's posture-dependent inertia matrix. The off-diagonal entries of $M$ matter: moving two coupled joints together can cost more or less than moving each one alone, which a flat (Euclidean) metric misses (Jaquier and Asfour; Klein et al.). We propose to build the analogous matrix for a sentence. The "inertia" comes from how sensitive a frozen multilingual language model's predictions are to changes in each word and in pairs of words. Once this matrix is computed for a sentence, the "switch energy" of any set of word substitutions is a cheap quadratic form. We then generate synthetic code-switched data by sampling low-energy substitution sets, using only a bilingual dictionary.
Core Claim
The claim is that a context-dependent coupling matrix, taken from a model's own predictive geometry, can tell natural switch sites from "tears" that leave the model's representation space. Three things should follow from it. First, the matrix's off-diagonal coupling should make it cheaper to switch a whole constituent (a grammatical phrase) than a single word inside it. That would recover phrase-like switching without a parser or a phrase table. Second, data sampled at low energy should keep the semantic structure that random substitution destroys. Third, it should match or beat alignment-based and attribution-based augmentation downstream.
The idea also offers a testable explanation of an open tension. Probes say random substitution breaks representations, yet Progressive Code-Switching and GLUECoS report that synthetic substitution helps zero-shot transfer. Our hypothesis is that the gains come from the low-energy part of the synthetic data, and the probe damage comes from the high-energy part.
The Metric
Let $e_i$ be the input embedding of word $i$ in a matrix-language sentence of length $n$. Let $\delta_i = e'_i - e_i$ be the displacement to the embedding of its dictionary translation. Let $J_{k,i}$ be the Jacobian of the model's masked-prediction logits at position $k$ with respect to $e_i$. Let $F_k$ be the Fisher information of the softmax at position $k$, which measures how sensitive the predicted distribution is to its logits. We define the coupling between candidate switch sites $i$ and $j$ as
$$C_{ij} = \sum_{k=1}^{n} (J_{k,i}\delta_i)^T F_k (J_{k,j}\delta_j).$$
This is the Fisher metric pulled back to word displacements. To first order it is the quadratic approximation of the KL divergence that a joint switch causes in the model's predictions. It plays the role of $M(q)$: it depends on context (the "posture") and has off-diagonal coupling between positions. For a substitution set $S$ with indicator vector $s \in \{0,1\}^n$, the switch energy is
$$E(S) = s^T C s.$$
Computing $C$ for one sentence takes $n$ forward-mode Jacobian-vector products, one per candidate site. After that, the energy of any of the $2^n$ subsets is free. This is the practical advantage over scoring each candidate sentence with a forward pass.
We then sample substitution sets from
$$p(S) \propto \exp(-E(S)/T)$$
at a fixed target switch rate. Because the energy is quadratic, this is an Ising-like model, and Gibbs sampling handles it easily. Raising the temperature $T$ over training gives a curriculum that is set by context-dependent coupling rather than by per-token attribution scores. Multi-word dictionary entries and SVO/SOV reordering stay outside the metric. We leave Gap 5 of the research file honestly unsolved and restrict substitution to dictionary entries.
Relation to Prior Work
The robotics work supplies the method, not the application. Jaquier et al. and Klein et al. use the inertia matrix as a Riemannian metric (a position-dependent way of measuring distance) for human and robot motion. Klein et al. report pointing joint-angle error of 0.047 against 0.188 for a Euclidean model. That is the evidence that coupling terms matter, and we test the same thing in language. Beik-Mohammadi et al. derive a metric from a learned decoder so that paths avoid regions with no demonstration data. Our pullback-Fisher metric does the same job for sentences: it keeps substitutions near what the model has seen. Riemannian metric learning via optimal transport and the kinetic-energy view of flow matching learn or use metrics only on continuous toy or generative spaces. We need no learned metric, because the language model already defines one.
On the language side, three lines of work motivate and constrain us. Code-Mixed Probes found the representation collapse but measured it with flat cosine similarity and rank correlations. We replace those with a metric that can be computed before any text is generated. Matrix Language Frame augmentation and SimAlign cs-5 enforce constituent structure through external parallel resources, which Lu et al. note are missing for dialects such as Taiwanese Hokkien. We try to recover that structure from coupling alone. Progressive Code-Switching orders substitutions with layer-wise relevance propagation, a technique that assigns each token one scalar importance score for a specific task. In our terms that is a diagonal, task-specific metric with no coupling. Our diagonal ablation reproduces that design inside our setup.
Research Program
The work has three stages, and each must succeed before the next is worth running.
The first stage is validation. On 2,000 sampled substitution sets per language pair, we check that the quadratic energy tracks the true KL divergence of predictions and the change in masked-LM pseudo-log-likelihood. We report Spearman correlation as a function of how many words are switched, since a first-order approximation should degrade as switches get larger.
The second stage is intrinsic evaluation. Natural code-switched corpora in LinCE (Aguilar et al.) and GLUECoS come with language-ID tags. We back-translate the embedded-language spans into the matrix language and ask whether the observed switch set has lower energy than random sets of the same size. We also ask whether the coupling term predicts that real switches fall on constituent boundaries. On the 316-post Spanglish probing set, we re-run the probes of Laureano De Leon et al. on low-energy and high-energy synthetic versions of each post.
The third stage is downstream training. We use XLM-R and mBERT encoders, and a small Transformer language model on SEAME.
Baselines
The naive reference points are, first, training with no augmentation and, second, uniform random dictionary substitution at the same switch rate. Every synthetic method receives the same dictionary, the same switch rate, the same token budget, and the same hyperparameter-search budget. We re-run every baseline in our own pipeline rather than comparing to numbers reported in other setups.
The strongest attribution-based competitor is Progressive Code-Switching (Li et al.). It is the best-reported dictionary-only method on our transfer benchmarks: PAWS-X average accuracy 88.9 with XLM-R, MLDoc average F1 84.7, and XTOD Thai slot F1 38.5, against 10.6 for vanilla mBERT. We run it with its own curriculum schedule. Beating it means a higher average on PAWS-X, MLDoc and XTOD across three seeds at equal compute. We count the relevance-propagation passes it needs against the Jacobian-vector products ours needs.
The strongest alignment-based competitors use parallel data our method does not get. On SEAME, Matrix Language Frame pretraining (Lee et al.) reports perplexity 173 against a 219 baseline. On KRCS, NLLB-3.3B fine-tuned with SimAlign cs-5 (Borisov et al.) reports BLEU 16.48 and COMET 0.56. Matching these numbers without parallel phrase tables would count as success. Beating them is not required.
For GLUECoS NLI the file gives a baseline accuracy of 63.1 (Khanuja et al.). No reported result was found on Lacuna for any metric-guided substitution method on GLUECoS or LinCE.
Two ablations isolate the key ingredient. The diagonal-only ablation keeps $C_{ii}$ and drops the coupling, which tests whether coupling matters. The flat-metric ablation replaces $C$ with $\delta_i^T\delta_j$, which tests whether the model's geometry matters at all.
Why not just…?
Why not just score candidate sentences by masked-LM pseudo-likelihood and keep the most fluent ones? This is the most serious threat. To first order our energy approximates the same quantity. The difference is cost and structure. Pseudo-likelihood needs a fresh set of forward passes for every candidate subset, so it cannot drive a sampler over all subsets of a sentence. Our matrix scores every subset after $n$ passes. We will run a pseudo-likelihood filter that sees the same number of candidates our sampler draws, at matched compute. If it matches downstream quality, our contribution shrinks to an efficient approximation, and we will report it that way.
Why not just use SimAlign or phrase-table alignment? These methods work, as the SEAME and KRCS results above show. But they need parallel text, which low-resource dialects lack (Lu et al.). The intrinsic constituent-boundary test, together with the head-to-head comparisons on SEAME and KRCS, will show how much of their benefit coupling recovers without parallel text.
Why not just use relevance-ordered progressive substitution? That method gives each token one task-specific scalar and ignores interactions between positions. The diagonal ablation is exactly that design inside our framework. If the diagonal ablation matches the full metric, coupling contributes nothing, and the robotics analogy fails for language.
Why not just prompt a large language model to write code-switched text? Yong et al. and Yoo et al. show this produces fluent text. Yoo et al. also find the models comprehend low-resource languages poorly. We found no reported downstream-augmentation number for prompted generation on these benchmarks. We will add a prompted-generation arm at equal token count for Spanish-English and Hindi-English. We expect it to be strong for these high-resource pairs, and our case for the metric rests mainly on the dictionary-only, low-resource setting.
Decisive Checks
Three outcomes would falsify the proposal. The first is if observed natural switch sets in LinCE are not lower in energy than random sets of the same size. The second is if the coupling term adds nothing over the diagonal. The third is if high-energy synthetic data does not reproduce the near-zero semantic-similarity correlation in the probing setup while low-energy data avoids it.
The proposal is confirmed if three things hold together. Low-energy data must keep probe correlations close to the natural 0.85–0.87 range. It must beat Progressive Code-Switching on the PAWS-X, MLDoc and XTOD average. And it must close most of the gap to Matrix Language Frame pretraining on SEAME perplexity without parallel data.
References
- Code-Mixed Probes Show How Pre-Trained Models Generalise on Code-Switched Text
- Linguistically Motivated Parallel Data Augmentation for Code-Switch Language Modeling
- Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair
- Riemannian Geometry as a Unifying Theory for Robot Motion Learning and Control
- A Riemannian Take on Human Motion Analysis and Retargeting
- Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching
- GLUECoS: An Evaluation Benchmark for Code-Switched NLP
- Learning Riemannian Manifolds for Geodesic Motion Skills
- Riemannian Metric Learning via Optimal Transport
- A Kinetic Energy Perspective of Flow Matching
- Exploring Methods for Building Dialects-Mandarin Code-Mixing Corpora: A Case Study in Taiwanese Hokkien
- LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation
- Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages
- Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding