Measuring What a Language Filter Cuts Off: Distance-Stratified Audits of English-Only Evidence

Many systematic reviews and bibliometric studies keep only English-language papers. Whether that distorts their conclusions is usually argued, not measured. This proposal treats a language filter as a cut through the citation graph, the network in which papers are nodes and citations are edges. It then borrows a tool from networked control, spatial correlation decay: the observation that in many networks, a node's influence on another shrinks roughly geometrically with the number of hops between them. In control this result reassures. An agent that looks only at nearby nodes loses little, because the distant ones barely affect its local goal. Evidence synthesis has the opposite goal. A pooled effect size or a global prevalence is an average over all nodes, so strong decay means the English side carries little information about what lies beyond the cut. That is exactly the case in which dropping the far side is most dangerous.

The practical result is a cheap audit procedure. Sort non-English studies by their citation-hop distance from the English set. Translate and code a small sample drawn from each distance band. Fit how the study-level outcomes drift as distance grows. Use that fit to estimate the bias of the English-only result, along with a rule for when further multilingual search is unlikely to change the conclusion. In two sentences: a language filter cuts the citation graph, and the bias it causes equals the evidence mass beyond the cut weighted by how far those studies have drifted. Because drift should grow with graph distance, sampling by distance band should estimate that bias with fewer translated papers than sampling at random.

Core Claim

Take any estimand computed from a body of studies, such as a share, a Gini coefficient, or a pooled effect. Let $\pi_d$ be the fraction of the full multilingual evidence that sits at citation distance $d$ from the English-included set, with $d=\infty$ for studies in components that never touch the English set. Let $\Delta_d$ be the mean difference in the study-level quantity between distance band $d$ and the English set. The bias of the English-only estimate is then

$$ \mathrm{Bias} = \sum_{d \ge 1} \pi_d\, \Delta_d + \pi_\infty \Delta_\infty . $$

This identity is trivial. The substantive hypothesis is about how $\Delta_d$ changes with $d$. Suppose each citation hop carries forward populations, methods, and reporting conventions with some persistence $\lambda \in (0,1)$, a stand-in for the $\gamma\rho(C)$ contraction factor in Local Policies for Graph-Structured Markov Decision Processes. Then a band's mean relaxes geometrically from the English mean toward the far cluster's own mean:

$$ \Delta_d \approx \Delta_{\max}\,(1-\lambda^{d}), $$

where $\Delta_{\max}$ is the asymptotic divergence of the far cluster. This model makes three testable predictions. First, divergence rises with hop distance and then saturates. Second, a few audited studies per band are enough to fit $\lambda$ and $\Delta_{\max}$. Third, the bias left after expanding the search to radius $m$ falls off like $\lambda^m$ times the remaining mass, which gives a stopping rule analogous to the $\mathcal{O}((\gamma\rho(C))^m)$ truncation bound. The disconnected stratum $d=\infty$ is the honest failure mode. Graph structure cannot extrapolate into it, so it must be audited directly. Its mass $\pi_\infty$ is reported as an explicit measure of how much the method cannot reach.

Why the Two Directions Belong Together

The decay literature supplies the mechanism, but its conclusions have to be inverted. Guannan Qu et al., Alex DeWeese et al., and Local Policies for Graph-Structured Markov Decision Processes show that policies restricted to a $\kappa$-hop neighborhood lose an exponentially small amount relative to the global optimum. They prove this for local objectives, and the third paper ties the rate to the spectral radius of a pairwise influence matrix. The most relevant result for this proposal is Fabio Pasqualetti et al., who show that in power networks, the error of estimating a node's state from local measurements depends on the node's distance from the observer. That observability view is the one that transfers. An English-side observer estimating a distant node is in the same position, and nothing in the decay results licenses dropping that node from a global average.

The publication-bias literature supplies the motivation and the testbeds. Weiwei Jin et al. argue that English-only surgical meta-analyses miss large cohorts from high-volume international centers. Jie Qian et al. find that eating-disorder prevalence syntheses are effectively limited to English and Chinese. Navin Kumar et al. note that local-language interventions are lost under English-only criteria. None of these papers says how much is lost or when the loss is small. Geographic Citation Gaps in NLP Research documents regional citation silos: North America and Western Europe produce 65% of ACL papers, and the citation gap between US papers and papers from India or Spain widens from 3.5x to 23x over 15 years. That is descriptive evidence that the graph is modular, which is the condition under which band-wise drift should be visible. That paper does not model diffusion across those silos, and the present proposal does.

The closest existing work is Automated Analysis Framework for Multilingual Climate-Health Literature Based on Multi-Agent Large Language Model. It extracts city, exposure, and outcome fields from 28,150 English and 4,492 Chinese papers with F1 0.92, and reports a geographic Gini of 0.761. It treats language and geography as attributes to extract. It never asks how the English-only answer would differ, or how one could predict that difference without coding the whole Chinese corpus. This proposal reuses its corpus design and extraction method as the ground-truth generator. ResearchTown: Simulator of Human Research Community also uses local message passing over a paper–author graph. Its graph is monolingual, and it makes no claims about errors caused by truncation, so it is related only in spirit.

Research Program

The first component is graph construction. For each review topic, the English-included set is whatever an English-only protocol would retain. Non-English papers on the same topic are pulled from regional indexes such as CNKI and Baidu Scholar, following the climate-health pipeline. Cross-language citation edges come from reference lists in either direction. Each non-English study gets its shortest hop distance to the English set, or $\infty$ if it is unreachable. The empirical decay of the citation graph itself is measured by plotting the probability that a finding appearing at distance $d$ is later cited on the English side, then fitting a geometric rate. This directly tests Gap 2 of the research file: does cross-lingual diffusion decay geometrically at all?

The second component is the distance-stratified audit. Given a translation and coding budget of $B$ papers, it allocates samples across distance bands, including a mandatory share for $d=\infty$. It codes those papers with the climate-health multi-agent extractor, fits $\lambda$ and $\Delta_{\max}$ by weighted least squares, and returns a bias estimate with a bootstrap interval. The allocation rule is Neyman-style: bands get more samples when their mass is larger and their within-band variance is higher, using variances from a small pilot.

The third component is a stopping rule. A reviewer expanding the search outward hop by hop stops at radius $m$ once the estimated residual $\sum_{d>m}\pi_d|\hat\Delta_d| + \pi_\infty|\hat\Delta_\infty|$ falls below a tolerance set in the units of the estimand. AI-Accelerated Design of Evidence Synthesis for Global Development explicitly names "when to stop" as an unresolved problem for active-learning screening. This rule is a candidate answer, but only for the bias question, not for recall.

Baselines

The naive reference point is the English-only estimate reported as if it were the global answer, which implicitly sets the bias estimate to zero. Every method must beat this. If the true bias is near zero on most topics, the whole program reduces to a reassuring negative result, and that outcome would be worth publishing.

The mass-only correction reports the share of non-English studies and flags reviews where that share exceeds a threshold. This is the most common informal practice. It will be scored on how well $\pi_{\text{non-English}}$ ranks topics by true bias. Beating it means showing that divergence, not just mass, accounts for a meaningful part of the variance in true bias across topics.

The strongest competitor for the bias-estimation question is a uniform random audit. It translates and codes $B$ non-English papers drawn at random and estimates $\Delta$ by a simple difference in means. It is unbiased, needs no graph, and any careful reviewer would propose it. It gets the same $B$, the same extractor, and the same bootstrap intervals. I found no reported result on Lacuna for uniform random language audits of systematic reviews. The proposal beats it if, at equal $B$, the stratified estimator has lower mean absolute error in estimated bias across topics, with nominal interval coverage, and needs fewer audited papers to reach a fixed error. A reasonable bar is 30% fewer audited papers at matched error.

For the retrieval side, the strongest existing method is BERT screening with Highest Priority active learning, as in AI-Accelerated Design of Evidence Synthesis for Global Development. On the 3ie Development Evidence Portal it reports a 68.5% reduction in screening effort at 80% recall over random screening with BERT alone, and 78.3% with active learning. We will run it with a multilingual encoder, seeded with English relevant papers, under the same labeling budget. Its output, the set of relevant non-English papers it found, will be plugged into the bias identity, with the found set treated as if it were the full set. TrialMind's search recall of 0.711–0.834 on TrialReviewBench, from Accelerating clinical evidence synthesis with large language models, is cited as the English-only retrieval ceiling. TrialReviewBench contains no non-English studies, so it cannot serve as a bias testbed.

Two ablations isolate the key ingredient. The first stratifies by text-embedding distance to the English centroid instead of citation-hop distance, which tests whether graph structure adds anything beyond semantic similarity. The second drops the decay-curve fit and uses per-band means only, which tests whether the geometric functional form matters.

Why not just…?

Why not just translate the queries and search every regional database? If a team can afford a full multilingual search and full coding, it should do that, and this proposal adds nothing to its estimate. The climate-health study shows that this route is not cheap. Standalone LLM extraction started at F1 0.66 and needed a multi-agent verification pipeline with gazetteers to reach 0.94 (Automated Analysis Framework for Multilingual Climate-Health Literature Based on Multi-Agent Large Language Model). The target user is a review team with a budget for tens of translated papers, not thousands, who needs to know whether that budget is enough. If coding costs keep falling, this concern weakens. The stopping rule would then still decide how far outward to search.

Why not just run multilingual active learning? Active learning answers a different question: which papers are relevant, not how much they change the answer. It also starts from English seed papers and ranks by similarity to them. That pushes it toward the near bands, which, if the drift hypothesis holds, are the papers least likely to change the estimate. The experiment measures this directly by reporting the hop-distance distribution of papers found by active learning and the resulting bias estimate. If active learning recovers the far bands and the disconnected stratum as well as distance stratification does, that would be a real refutation, and the audit should then be built on top of active learning rather than replacing it.

Why not just audit a random sample? This is the most serious threat. A random audit is unbiased and assumption-free, and if drift does not depend on distance, it will match or beat the stratified design. The head-to-head comparison at equal budget settles the question. The proposal claims an advantage only if hop distance explains a substantial share of between-study heterogeneity. That share is measured, not assumed.

Why not just give an LLM all the abstracts and ask? Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias found that GPT-4's suggested references had a median citation count 1,326 higher than the ground-truth references. That is a popularity pull that would push estimates toward the dense English core. A long-context LLM summary will be included as an extra estimator so this concern is measured in this setting, not inferred from a different task.

Proposed Experiments

The primary testbed rebuilds the bilingual climate-health corpus following the published protocol: OpenAlex for English and CNKI/Baidu Scholar for Chinese, coded with the multi-agent extractor and checked against an 800-paper gold set in the same style. For each exposure–outcome topic with enough studies, the ground truth is the bilingual value of three estimands: the share of studies reporting an adverse association, the city-coverage Gini (the reported bilingual figure of 0.761 is a sanity check), and the distribution of studied regions. The true bias is the English-only value minus the bilingual value. Each method then receives budgets $B \in \{10, 25, 50, 100\}$ translated papers per topic, and we report mean absolute error of estimated bias, 90% interval coverage, and the budget each method needs to reach a fixed error.

The secondary testbed is geographic rather than linguistic. It uses the ACL Anthology citation data from Geographic Citation Gaps in NLP Research, truncated to North American and European affiliations. The estimands are topic shares and citation-based rankings. This tests whether the decay-and-drift pattern holds for a regional cut, where the ground truth is complete. A stretch clinical case rebuilds the robotic versus laparoscopic pancreatoduodenectomy comparison discussed by Weiwei Jin et al. with Chinese-language cohorts added, and pools a real effect size, the setting where reviewers care most.

Three checks are decisive. First, the fitted cross-lingual citation decay should be roughly geometric, and $\Delta_d$ should rise with $d$ and level off. If drift is flat in distance, the core claim fails, and the right conclusion is that random audits are sufficient. Second, the stratified estimator should beat both the uniform random audit and the text-distance ablation at equal budget. If the text-distance ablation matches it, the graph is unnecessary. Third, among topics where the stopping rule declares the search complete, the true residual bias should fall within tolerance at close to the nominal rate. Throughout, $\pi_\infty$ is reported alongside every result, so readers can see how much evidence lies beyond the reach of any citation-based method.