Expert panels increasingly decide what counts as a working treatment when trial evidence is thin. More and more, an AI system first summarizes the literature for these panels. Those summaries are drawn almost entirely from English-language databases. We propose to measure, with a randomized experiment, whether that restriction changes what experts conclude. The test case is plant-derived alternatives to antibiotic drugs for coccidiosis, a parasitic gut disease of chickens that costs poultry producers through lost growth and wasted feed. Much of the trial literature on these products is published in Chinese, Portuguese and Spanish.
The design has two arms. In both, matched expert panels rate the same statements about which products work and what "working" should mean. One arm receives an English-only evidence summary. The other receives a summary that also draws on non-English trials. If the panels' conclusions diverge, then language restriction is changing decisions and not only bibliographies. That would make the multilingual evidence pipeline a necessary part of AI-assisted consensus methods rather than an optional courtesy.
Core Claim
The claim is testable. For a decision domain where a large share of primary trials is non-English, the language coverage of the summary given to an expert panel will shift two things: pooled effect estimates, and the panel's consensus on a meaningful fraction of statements. The shift will be larger for production endpoints that carry economic weight than for parasite-count endpoints. The second half follows from where the evidence sits. Regional broiler trials typically report weight gain and feed conversion ratio, the kilograms of feed needed per kilogram of bird weight gained. Lab-oriented English studies more often report oocyst counts, the counts of parasite eggs shed in feces.
The new element is the causal manipulation of evidence language inside a consensus process. Existing studies either document English-only bias in reviews or build English-only AI review tools. None measures the downstream effect on expert decisions.
Why This Domain, and the Role of Each Direction
Economic impact of poultry disease is the application. Coccidiosis, caused by Eimeria parasites, destroys intestinal tissue, impairs nutrient absorption and worsens feed conversion. The cost is mainly chronic margin erosion, not deaths. Botanical products show "moderate" activity against E. tenella with high safety margins in English-language broiler trials, for example a eucalyptus–apigenin–eugenol formulation and eugenol-containing essential oils. Those trials are the English slice of a much larger, regionally published literature, so the domain has a real decision stake and a plausible language skew.
The disease-modifying therapy gap is the conceptual question the panel must settle. In human joint disease, the field distinguishes drugs that mask symptoms from drugs that change the underlying tissue damage. Ying et al. frame joint replacement as relieving pain without touching the drivers of degradation. Multi-component herbal preparations such as Osteoking in a mouse model are credited with structural effects because they reverse cartilage and bone changes through a mechanism that a gene knockout abolishes.
Poultry anticoccidials have an analogous, unstated distinction. Some products suppress oocyst shedding. Others restore gut function and growth, which is where the money is. No agreed definition separates the two. The delayed-start trial design from Alzheimer's research provides a candidate operational criterion that the panel can rate. In that design, if late starters never catch up, the effect is structural rather than symptomatic.
One finding constrains which endpoints panels should trust. Automated image analysis of coccidia found that automated oocyst counting is unreliable for small species such as E. acervulina. In multi-species samples, 92.32% of counts fell outside the 95% confidence interval. Oocyst counts are therefore a shaky sole endpoint, which strengthens the case for production outcomes.
The Delphi method is the decision instrument. A Delphi study runs anonymous, repeated rating rounds with aggregated feedback between rounds. The ankle cartilage consensus shows the method producing 12 recommendations, 10 at "Strong Consensus" (75–99% agreement), from Grade E evidence. It also shows that such panels never audit the language coverage of the literature they rely on.
The Human-AI Hybrid Delphi model supplies the method we reuse. A large language model writes an "evidence floor", a literature synthesis shown to experts before they rate. Compact six-person panels then reach consensus on 92.3% of items in one round. Its corpus was curated in English. We keep its architecture and vary only the language coverage of the floor.
Linguistic publication bias is the manipulated variable. Prior work motivates the question but does not answer it. Geographic citation gaps in NLP quantifies who gets indexed and cited. The antibiotic-discovery knowledge graph attributes missed compounds to its English-only PubMed scope. AgentSLR and Bio-SIEVE automate reviews while acknowledging that their pipelines are English-only. All four identify the risk. None measures its effect on a decision.
Research Program
Stage one: build the evidence base and a gold standard. We assemble the full trial literature on non-antibiotic (botanical, essential-oil and probiotic) coccidiosis control in broilers. Sources are English indexes plus Chinese, Portuguese and Spanish regional databases and theses. Native-speaker veterinary annotators label eligibility and extract a fixed record for each trial. The record covers challenge model, Eimeria species, dose, comparator, weight gain, feed conversion ratio, lesion score, oocyst output, mortality, and a risk-of-bias rating.
This gold set is the first multilingual veterinary evidence-extraction benchmark we know of. No reported result was found on Lacuna for multilingual veterinary trial extraction. The annotation scheme extends the preclinical animal schema of PICO entity extraction for animal literature. PICO is the standard record of population, intervention, comparator and outcome. Their schema adds species, strain and disease-induction fields.
Stage two: measure whether language shifts the evidence. We run random-effects meta-analyses on feed conversion ratio, weight gain, lesion score and oocyst output. Each is run twice, once on English-only trials and once on all trials. Because non-English trials may differ in quality, each comparison is repeated after restricting to low-risk-of-bias trials. This separates a language effect from a quality effect.
Stage three: measure whether the pipeline can recover non-English evidence. We compare automated screening and extraction pipelines against the gold set, broken down by language.
Stage four: measure whether shifted evidence shifts decisions. We recruit roughly 48 poultry veterinarians, nutritionists and parasitologists, balanced by region. They are randomly assigned to eight six-person panels, following HAH-Delphi's finding that six senior experts reach thematic saturation. Four panels receive an English-only evidence floor and four a multilingual floor.
The two floors are generated by the same model with identical prompts, length and format. Every claim in either floor carries a citation that can be checked against the gold extraction, so hallucinations can be counted. All panels rate the same roughly 40 statements over two rounds. The statements cover product efficacy, endpoint priority, and candidate definitions of "disease-modifying," including the delayed-start criterion.
Baselines
The naive reference is the English-only status quo. Human reviewers screen PubMed and Web of Science results and hand an English synthesis to the panel. This is the condition most consensus statements are produced under today. Any multilingual effect is measured relative to it, on the same statements and the same expert pool.
The strongest existing pipeline is AgentSLR, an agentic workflow that retrieves, OCR-converts, screens full texts and extracts parameters into a schema. AgentSLR reports 0.92 recall for identifying relevant parameters, and full-text screening raised screening recall from 0.81 to 0.89 (AgentSLR). For screening accuracy, the reference is Bio-SIEVE's instruction-tuned Guanaco-7B, which reports 0.82 accuracy against 0.60 for zero-shot ChatGPT (Bio-SIEVE). For animal-study extraction, PubMedBERT with self-training reports 71% overall NER F1, the harmonic mean of precision and recall for tagging entities in text. Comparator F1 reaches only 48.5% (PICO for animal literature).
We run AgentSLR's open-weight configuration (gpt-oss-120b) unchanged on our corpus, with the same retrieval budget and compute as our variant. A multilingual pipeline "beats" it if two conditions hold. First, its recall on non-English eligible trials is within five points of its recall on English trials. Second, its pooled-effect estimates fall inside the gold-standard confidence intervals, on endpoints where the AgentSLR-translated version falls outside them.
The consensus baseline is HAH-Delphi itself, run on the English-only floor. It reports 95% directional agreement with a six-expert insomnia panel (HAH-Delphi). Our multilingual floor is judged better if it moves panel ratings toward the gold-standard meta-analysis, rather than merely changing them.
Two ablations isolate the key ingredient. The first is translate-then-extract. Non-English papers are machine-translated and fed to the English pipeline, which tests whether native-language extraction is needed at all. The second is a size-matched English floor. The English-only floor is padded to the same number of trials using additional lower-ranked English studies, which separates "more evidence" from "different-language evidence."
Why not just…?
Why not just compare English-only and multilingual meta-analyses and skip the expert panels? This is the most serious alternative, and stage two is exactly that analysis. If language shifts pooled effects, that result stands alone. However, the panel statements include questions a pooled effect cannot answer, such as which endpoints define disease modification and how low-quality regional trials should be weighted. We will report how often a shifted pooled effect fails to shift consensus, and the reverse.
If shifts in consensus are fully predicted by shifts in pooled effects, the panel layer adds nothing. In that case we will say the cheaper meta-epidemiological check is sufficient. That outcome is a legitimate finding, not a failure.
Why not just machine-translate everything into English and use existing tools? This may well suffice. The translate-then-extract ablation tests it directly against the native-speaker gold set. Two pieces of evidence suggest it may fall short. The antibiotic-discovery work attributes errors to unstable nomenclature across sources and to the need to link findings across separate papers (Delmas et al.). Comparator and dose fields are already the weakest even in English (PICO for animal literature). If translation matches native extraction within a few points, we adopt it as the recommended pipeline.
Why not just let the language model produce the consensus without human experts? HAH-Delphi's 95% replication was measured on English items that already appear in the literature. We will have the model rate all 40 statements from each floor and compare its ratings with the human panels' ratings. Disagreement concentrated on definitional and economic statements would support HAH-Delphi's claim that experiential judgment is irreplaceable. Uniform agreement would mean the human panels add little in this setting, and we would report that.
Proposed Experiments and Decisive Checks
The first check is prevalence. We report what fraction of eligible trials is non-English and how much the English-only and multilingual pooled estimates of feed conversion ratio and lesion score differ. Both are reported before and after risk-of-bias restriction.
The second check is extraction quality by language. Measures are recall and field-level F1 against the gold set for AgentSLR, translate-then-extract, and native multilingual extraction, plus the hallucination rate in each evidence floor.
The third and decisive check is the panel experiment. The primary outcome is the proportion of statements whose consensus tier differs between arms. Tiers follow the ankle consensus cutoffs: Consensus 51–74%, Strong 75–99%, Unanimous 100%. The analysis uses a statement-level permutation test that respects panel clustering. The secondary outcome is whether multilingual-arm ratings move closer to the gold meta-analysis than English-arm ratings.
The prediction is that divergence concentrates on production-endpoint statements. If it instead appears on oocyst-count statements, our reasoning about where regional evidence lies is wrong.
With eight panels, power to detect small shifts is limited, so we preregister the statement set and the analysis. If neither arm differs and pooled effects do not shift either, that is strong evidence that English-only consensus is safe in a domain where it was most likely to fail.
References
- Anticoccidial activity of a botanical natural product based on eucalyptus, apigenin and eugenol against Eimeria tenella in broiler chickens
- Anticoccidial activity of essential oils containing eugenol against Eimeria tenella in broiler chickens
- Case Report: First Documented Hip Arthroplasty on Chinese Patient with Ochronotic Arthropathy
- Osteoking Decelerates Cartilage Degeneration in DMM-Induced Osteoarthritic Mice Model Through TGF-β/smad-dependent Manner
- A Novel Approach to Delayed-Start Analyses for Demonstrating Disease-Modifying Effects in Alzheimer’s Disease
- Automated Image Analysis for Detection of Coccidia in Poultry
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- Geographic Citation Gaps in NLP Research
- Accelerating Antibiotic Discovery with Large Language Models and Knowledge Graphs
- AgentSLR: Automating Systematic Literature Reviews in Epidemiology with Agentic AI
- Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
- PICO Entity Extraction For Preclinical Animal Literature