A multilingual model fine-tuned only on English and then tested on German or Hindi does worse in Hindi, and the usual explanation is made at the level of whole languages. The explanation says Hindi is typologically far from English, is written in another script, or had less pretraining data. Yet within any single target language, some sentences transfer almost perfectly and others fail badly. This proposal asks whether that within-language variation has a linguistic signature. Concretely, it tests whether zero-shot failures concentrate on test sentences whose measurable structure is far from the structure of the source-language training data. Examples of such structure are head-final clauses, dropped subjects, heavy inflection per word, and long dependencies. It then tests whether the signature moves when the source language changes. If it does, "English-centric bias" stops being a slogan about benchmarks. It becomes a measurable, sentence-level property: the model has learned the task in the shape of its source language.
The tool is multi-dimensional linguistic profiling, meaning a vector of interpretable text measurements such as sentence length, part-of-speech mix, parse depth, dependency distance, and morphological feature counts. So far this kind of profiling has been used almost only on English, and only to explain monolingual model behavior. We rebuild it on Universal Dependencies annotation so that one feature means the same thing in every language. We then use it to explain the part of a sentence's error that is specific to transfer, not the sentence's overall difficulty.
Core Claim
The central quantity is the transfer-specific error of a test instance $x$ in target language $t$. It is the zero-shot model's loss on $x$ minus the loss of a model of the same architecture trained with in-language supervision for $t$, averaged over seeds. Subtracting the in-language loss removes difficulty that any model would face: long sentences, rare words, ambiguous labels. What remains is the cost of having learned from another language.
We claim three things about this quantity. First, a meaningful share of its variance lies within languages rather than between them, so language-level typology cannot capture it. Second, the distance of $x$'s linguistic profile from the source training distribution predicts it better than generic difficulty proxies. These proxies are length, subword fertility (tokens per word), and the model's own confidence. Third, the features carrying this effect are source-relative. When the model is fine-tuned on German or Japanese instead of English, the predictive features and the direction of their effect should change to track the new source's profile. The third claim is the one that distinguishes a language-bias account from a "some sentences are just hard for multilingual encoders" account.
Relation to Prior Work
The closest work explains transfer at the level of languages. Evaluating morphological typology in zero-shot cross-lingual transfer groups 19 languages into four morphological classes. It shows that POS transfer within the fusional group exceeds 80% while introflexive targets fall below 50%, and that pretraining presence and typological match reduce transfer loss by 25.9 and 8.6 points. Untangling the Influence of Typology, Data, and Model Architecture on Ranking Transfer Languages for Cross-Lingual POS Tagging shows that fine-grained Grambank and URIEL vectors combined with dataset features rank source languages much better than coarse distances. Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks adds pragmatic features, such as a pronoun-and-verb ratio as a proxy for high-context languages, to rank sources for sentiment. All three predict one number per language pair. We predict one number per test sentence, and we treat language identity as a nuisance variable to be absorbed rather than the object of study. Their findings motivate our feature choices: morphology matters, fine-grained features beat coarse ones, and pronoun drop matters for some tasks. Their unit of analysis is exactly what we move beyond.
The profiling methods come from monolingual diagnostics. Text Characterization Toolkit (TCT) extracts 61 English text features and fits meta-models that find subsets where a model's accuracy falls from 0.90 to 0.40. What Makes My Model Perplexed? predicts BERT and GPT-2 sentence perplexity from 78 UD-derived features, with R² of 0.38 and 0.22, and warns that the relation is correlational and length-sensitive. We reuse their instance-level meta-model design and their UD-based feature families. We drop the English-only lexical and psycholinguistic resources, because no comparable norms exist across our target languages; we say plainly that the psycholinguistic stratum is out of scope. We take the correlational warning seriously, which is why the program includes source swaps and targeted rewrites rather than regression alone.
Two lines of work constrain the design. A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank finds that English is a weak donor once data size is equalized, and that shared script predicts transfer better than language family. We therefore control source training size across source languages and include script and subword-fertility features explicitly, so structural effects are not confused with volume or tokenization. Generalization Measures for Zero-Shot Cross-Lingual Transfer shows that source-side prediction margin correlates up to 0.954 with target accuracy across training runs, and that loss sharpness correlates up to −0.845. Those correlations are across models, not across sentences. Still, a per-instance analogue, the model's own confidence on $x$, is a serious competing predictor, and we treat it as one.
On language bias, Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis shows that zero-shot transfer amplifies racial and gender bias even when task F1 matches monolingual models. It also finds that grammatical-gender languages behave differently. Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques finds that English-side debiasing overcorrects and that grammatical gender interferes with debiasing projections. Neither says which sentence structures carry the imported bias. We use their counterfactual template methodology in one secondary experiment.
Research Program
The first component is a cross-lingual profiler. Every test sentence is parsed with a single multilingual UD model, following the setup of 75 Languages, 1 Model: Parsing Universal Dependencies Universally, so annotations are consistent across languages. From the parse we compute roughly forty language-neutral features, grouped into four families. Surface features are length, character and subword counts, and fertility. Morphosyntactic features are the UPOS distribution, morphological features per token, and case and agreement marking rates. Syntactic features are parse depth, mean and maximum dependency distance, the proportion of head-final arcs, and clause counts. Discourse and pragmatic features are overt-subject rate, pronoun ratio in the spirit of the context-level ratio from the cross-cultural paper, and connective counts. To measure parser noise, we compute the same features from gold and predicted parses on UD test sets and report their agreement. Any feature whose predicted values are unreliable in a given language is excluded for that language.
The second component is the source-relative distance. The source fine-tuning set is profiled the same way. For each test sentence we compute the feature-wise standardized deviation from the source distribution, plus a single Mahalanobis distance in profile space. The hypothesis is that these distances, not the raw feature values, predict transfer-specific error.
The third component is the meta-model and its decomposition. Transfer-specific error is regressed on three groups of predictors: a language fixed effect, the difficulty proxies (length, fertility, zero-shot confidence), and the profile features, both raw and source-relative. We use nested models and hold out whole languages, and we report how much within-language variance the profile adds beyond language identity and difficulty proxies. Both a linear model and gradient-boosted trees are fit so that nonlinearity does not decide the result.
The fourth component is a pair of interventions. In the source swap, the same model is fine-tuned on size-matched data from English, German, and Japanese for tasks where native source data exists. The meta-model is refit for each source, and we test whether the sign and rank of feature effects move with the source profile. For example, head-final proportion should hurt under an English source and help under a Japanese source when the target is Korean or Turkish. In the targeted rewrites, native speakers make meaning-preserving edits to about 300 sentences per language that change one feature, such as restoring a dropped subject in Japanese or splitting an embedded clause in Hindi. We check whether transfer-specific error falls more for the zero-shot model than for the in-language model.
Baselines
The naive reference point is the language-mean model. It predicts each sentence's transfer-specific error as the average for its language, which is the implicit assumption of all language-level work. Any profile-based predictor must add held-out within-language explained variance on top of it. Otherwise the claim fails.
The strongest competitor for predicting instance errors is a difficulty-only meta-model using length, subword fertility, and the zero-shot model's own prediction margin on $x$. Margin is the per-instance version of the signal that Generalization Measures for Zero-Shot Cross-Lingual Transfer found correlates 0.954 with accuracy across models. Length is the baseline that What Makes My Model Perplexed? had to beat. This competitor gets the same training instances, the same held-out-language protocol, and the same hyperparameter search budget as the profile model. Beating it means a statistically reliable increase in held-out R² on transfer-specific error. Significance is tested by bootstrap over sentences, stratified by language. We also require an improvement in AUROC for flagging the top-decile transfer failures. No reported result was found on Lacuna for instance-level transfer-error prediction. The nearest reported figures are the English monolingual ones: the TCT failure subsets (0.90 to 0.40 accuracy) and the perplexity R² of 0.38 for BERT.
The strongest language-level method is a typology model. It uses Grambank vectors and dataset features as in Untangling the Influence of Typology..., expanded to every instance of the language. It serves as the best version of "typology explains transfer" and is evaluated identically.
The key ablation removes the source-relative distances and keeps the raw profile features. If raw features do as well, the effect is sentence difficulty rather than distance from the source, and the language-bias interpretation fails. A second ablation replaces transfer-specific error with raw zero-shot error. It shows how much of any naive profile-predicts-failure result is really generic difficulty.
For the transfer systems themselves, we fine-tune mBERT and XLM-R base with a fixed recipe. We report their target accuracy next to published numbers so the setup is not unusually weak. Published figures are XLM-R's 80.9% average XNLI accuracy, 14.6 points above mBERT, from Unsupervised Cross-lingual Representation Learning at Scale, and the POS transfer ranges from Evaluating morphological typology in zero-shot cross-lingual transfer.
Why not just…?
Why not just use language-level typology, since it already predicts transfer well? Typology assigns every sentence in a language the same prediction, so by construction it explains none of the within-language variance. The decomposition measures exactly how large that variance is. If within-language variance turns out to be small, say under a fifth of the total after removing noise from seed averaging, then typology is enough for practical purposes. We would report that outcome as a negative result.
Why not just use the model's own confidence and sentence length to flag risky inputs? This is the real threat. Confidence may already encode "this looks unlike my training data," which would make profiling redundant for flagging failures. The difficulty-only baseline tests this directly. Even if confidence matches the profile model on flagging accuracy, it cannot say which structure is responsible or predict how a source swap changes the pattern. So in that case we would drop the claim to a better failure detector and keep only the explanatory and causal claims, which rest on the source swap and the rewrites.
Why not just translate the test set into English and skip the problem? Translate-test changes the question rather than answering it. It also has documented costs: Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model found machine-translated training underperformed direct zero-shot transfer on Chinese, and XNLI: Evaluating Cross-lingual Sentence Representations reports translate-test is limited by translation quality for low-resource languages. We do use it as a probe. If translation removes transfer-specific error mainly on high-distance sentences, that supports the source-relative account.
Why not just flatten the loss landscape with sharpness-aware training? Flat minima improve average transfer, but they could do so uniformly or selectively. We train one SAM variant and test whether it shrinks the slope linking source-profile distance to error. A shrinking slope would be the first evidence connecting optimization geometry to linguistic structure. An unchanged slope would mean flatness helps along a different axis than the one profiling measures.
Proposed Experiments
The main study uses tasks that have genuine in-language supervision, since transfer-specific error needs an in-language reference. The first is UD POS tagging and dependency parsing on roughly twenty target languages that cover the four morphological classes and at least five scripts. Native English, German, and Japanese treebanks serve as sources, subsampled to equal size. The second is XNLI across its fifteen languages, with translate-train models as the in-language reference. Because translated training data carries its own artifacts, we confirm conclusions against the UD results rather than relying on XNLI alone. The third is multilingual sentiment on the SemEval-2016 and Amazon review sets used in Cross-Cultural Similarity Features, which have native training data. Metrics are held-out-language R² and top-decile AUROC for transfer-specific error, the within/between variance decomposition, and standardized feature coefficients for each source language.
The secondary bias experiment reuses the counterfactual template design of Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis for Japanese, Chinese, Spanish, and German. It varies template syntax, for example with and without an overt subject, or with and without an embedded clause, and tests whether the gap in bias between zero-shot and in-language models grows with source-profile distance. A positive result would give the first structural account of which sentences carry imported bias. A null result would suggest bias importation is lexical rather than structural.
Three results would settle the proposal. The first is whether source-relative profile features add held-out within-language explained variance beyond both the difficulty-only baseline and the typology baseline. The second is whether feature effects reverse or re-rank when the source changes from English to Japanese at matched data size. This is the check that separates learned source bias from intrinsic difficulty, and it has no counterpart in prior work. The third is whether single-feature rewrites reduce the zero-shot model's transfer-specific error more than the in-language model's. If the first holds but the second does not, the profiler is a useful failure detector but the language-bias interpretation is wrong, and we would write it up as such.
References
- Evaluating morphological typology in zero-shot cross-lingual transfer
- Untangling the Influence of Typology, Data, and Model Architecture on Ranking Transfer Languages for Cross-Lingual POS Tagging
- Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks
- Text Characterization Toolkit (TCT)
- What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity
- A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank
- Generalization Measures for Zero-Shot Cross-Lingual Transfer
- Cross-lingual Transfer Can Worsen Bias in Sentiment Analysis
- Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques
- 75 Languages, 1 Model: Parsing Universal Dependencies Universally
- Unsupervised Cross-lingual Representation Learning at Scale
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- XNLI: Evaluating Cross-lingual Sentence Representations