Retrospective cohort studies estimate treatment effects from records collected during routine care, where no one randomized who got which treatment. Their best-known failure is confounding by indication: doctors give a treatment because of how sick or robust a patient already is, so the treated and untreated groups differ before treatment begins. A retrospective study of an aggressive surgery against watchful waiting can then report a dramatic survival benefit that mostly reflects which patients were fit enough for surgery. This proposal asks whether those inflated findings are cited more than comparable, more cautious studies of the same clinical question. It also asks whether the geographic citation gaps documented in bibliometrics shrink, persist, or reverse once we compare papers that answer the same question with the same design quality.
The idea is to treat the citation record itself as a retrospective cohort and analyze it with the epidemiologist's tools. The unit of study is a published retrospective study. The "exposure" is how exposed its headline estimate is to confounding by indication. The "outcome" is its citations over five years. The comparison is made only among papers that address the same clinical question in the same era. The literature's own citation choices have their own confounders: topic, journal prestige, and author country. Question-matched sets are how we handle them, in the same way a careful cohort study handles patient prognosis.
Core Claim
Our hypothesis is that, within a fixed clinical question, retrospective studies with high risk of confounding by indication accrue more citations than lower-risk studies of the same question. We expect this because such studies tend to report larger favorable effects, and citing authors respond to effect size rather than to design. We further hypothesize that part of the per-paper citation gap between high-volume, lower-cited countries and high-citation countries is explained by differences in question choice and reported effect size. We expect the rest to remain after that matching.
The first claim is testable and would be new. The file records no study linking a paper's vulnerability to indication bias with its downstream citations. The second claim turns the descriptive gaps of Geographic Citation Gaps in NLP Research and the WoSCC bibliometric studies into a decomposition we can falsify. We do not claim to identify a manipulable causal effect of "being confounded". A paper cannot be assigned its design. The estimand is a question-conditional, confounder-adjusted contrast, and we will say so plainly.
Why This Intersection
Three bodies of work motivate the question.
The first is clinical. Several retrospective studies show how channeling produces striking estimates. Long-Term Survival in Men With High Grade Prostate Cancer reported a 68% lower cancer-specific death rate for radical prostatectomy than for conservative care after propensity adjustment. Its authors conceded that younger, healthier men were steered to surgery and that comorbidity data were missing before 1988. Revisiting diabetes risk of olanzapine versus aripiprazole used targeted minimum loss-based estimation, a doubly robust estimator that combines an outcome model and a treatment model so that either being right suffices. It found olanzapine associated with lower diabetes risk (risk ratios 0.77 to 0.90) and attributed this to clinicians steering the drug to low-risk patients. A target trial emulation in VetCompass canine diarrhoea records balanced indication with inverse probability weighting and found essentially null effects (risk difference 0.4%, 95% CI −4.5% to 5.3%). These papers are the prototypes for our exposure rubric. They show the same design family yielding large, clinically exciting estimates or careful nulls depending on how channeling is handled.
The second is the limitations literature, which constrains what adjustment can do. Laparoscopic versus open pancreaticoduodenectomy stresses that propensity matching cannot see unrecorded frailty or surgeon judgment. Risk factors for steroid-refractory ICI colitis shows that record completeness itself tracks severity and affluence. 90-day mortality prediction in elective visceral surgery documents healthy patients dropping out of follow-up. These results constrain the design in two ways. We score exposure from what a paper did to address channeling, not from whether its estimate is "true". We also treat our own citation data as subject to the same informative-missingness problems.
The third is bibliometric, and it supplies the outcome and the disparity question. Geographic Citation Gaps in NLP Research showed that US papers at the same venue out-cite Chinese and Indian ones, with the gap growing from 3.5x to 23x over 15 years. The authors left its causes open, and venue stratification was their only control. The Elephant in the Room found higher time-normalized citations for industry labs but could not separate quality from brand. The two biomedical WoSCC studies, AI in colorectal cancer and ferroptosis in Parkinson's disease, report the same volume-versus-impact split between China and the United States. All four stop at descriptive tabulation. None conditions on what the paper actually found or how it was designed. That is the step we add.
Research Program
The work has three parts: build a corpus of question-matched sets, score each paper's exposure to indication bias, and estimate citation contrasts within sets.
We will build the corpus from the Web of Science Core Collection, supplemented by PubMed to reduce the single-database coverage problem that the WoSCC studies acknowledge. We retrieve articles self-described as retrospective cohort studies of a treatment comparison, published over a ten-year window. We keep only those with at least five full years of citation follow-up. This choice addresses the trailing-indicator problem and mirrors the fixed follow-up of a cohort study.
We then group papers into question sets. A question set is a group of papers comparing the same intervention and comparator in the same condition within a three-year publication window. Surgery versus conservative care for high-grade prostate cancer and laparoscopic versus open pancreaticoduodenectomy are two examples. A language model proposes the groupings from the population, intervention, comparator and outcome it extracts from abstracts. Two annotators with epidemiology training confirm them on a stratified sample. We only analyze sets containing papers from at least two countries.
Next we score exposure. For each paper, we extract, from full text where available, a confounding-by-indication vulnerability score. It has two components. The first is structural. Is the comparison between an intervention given by prognosis and one withheld by prognosis, as with surgery versus observation, rescue therapy, or contraindication-driven drug choice? The second is analytic. Did the paper adjust at all? Did it use regression, matching, weighting, doubly robust estimation, or an explicit target trial emulation? Did it report a sensitivity analysis for unmeasured confounding, such as an E-value, or a negative-control outcome? We also extract the headline effect and its direction, and the strength of the causal language in the abstract. We need causal language separately to distinguish "confounding attracts citations" from "confident wording attracts citations".
Human double-coding of a random 10% sample calibrates the extraction. We report agreement and propagate misclassification into the estimates. We treat affiliation-to-country mapping the same way, since Geographic Citation Gaps in NLP Research reports only 84.5% tagging accuracy.
The primary estimand is the difference in five-year log citations between high- and low-vulnerability papers within the same question set. It is adjusted for journal, publication year, author team size, open access status, author-country group, prior author citation record, and sample size. The main estimator is targeted minimum loss-based estimation with an ensemble of flexible learners, following the approach in Agniel et al.. The question set enters as a fixed effect. The mediation analysis then splits the contrast into a part flowing through reported effect size and causal language, and a direct remainder.
The disparity analysis reuses the same machinery. Author-country group becomes the exposure. We then ask how much of the raw US-versus-China per-paper gap among retrospective studies survives three successive levels of adjustment: venue stratification alone, as in Rungta et al.; then question matching; then design and effect-size adjustment.
Before trusting any of this on real data, we will check the estimators on a semi-synthetic citation benchmark built in the spirit of the Infant Health and Development Program (IHDP) benchmark used in Targeted Deep Architectures. IHDP keeps real covariates and simulates outcomes with a known effect. We will keep the real paper covariates and simulate citation counts with a planted vulnerability effect, a planted prestige confounder that is only partly observed, and informative loss of tracking for papers in regional journals. This lets us measure bias and interval coverage for every estimator we later use.
Baselines
The naive reference point is the raw difference in mean five-year citations between high- and low-vulnerability papers across the whole corpus. This is what a descriptive bibliometric table would report. It is the number we expect to be most distorted by topic, because some fields both run more channeled comparisons and cite more heavily. The naive and adjusted estimators run on identical data, so the only difference is adjustment. Beating the naive baseline is not a performance target. The comparison shows how much of the raw association topic confounding explains.
The strongest existing analytic method on the disparity question is the venue-stratified median comparison of Geographic Citation Gaps in NLP Research. That paper reports mean-citation gaps among top-10 countries rising from 6.31 to 15.11 over 21 years, and a US-to-India or Spain ratio growing from 3.5x to 23x over 15 years. Those numbers are for NLP, not clinical retrospective studies. No reported result was found on Lacuna for venue-stratified geographic citation gaps among retrospective clinical cohort studies, so we will compute this baseline ourselves on our corpus using their stratification.
Their approach is the strongest competitor because it is exactly what the field does now. It is also a plausible sufficient explanation: if venue captures most of what matters, question matching adds nothing. Question matching counts as a real advance only if its gap estimate differs from the venue-only estimate by more than the combined confidence interval. If the two agree, we will report that venue stratification suffices.
The strongest estimation baseline is multivariable negative binomial regression of citation counts with journal and year fixed effects. This is the standard adjusted citation model. Its natural competitor within our program is propensity score matching on the same covariates. The file documents how propensity matching fails when covariates are missing or unrecorded (Zhou et al.). Both run on the same covariate set, the same question sets, and the same outcome definition. The ensemble learners in the doubly robust estimator get a fixed, pre-registered candidate library, with no extra tuning on the real outcome.
On the semi-synthetic benchmark, the relevant published reference is that Targeted Deep Architectures reports 92–92.4% interval coverage on IHDP against 73.5% for a naive neural plug-in estimator. It also reports 92% coverage against 55.8% for Kaplan–Meier under informative censoring. Those numbers come from a different benchmark. On ours, the doubly robust estimator beats regression if its 95% intervals cover the planted effect at least 90% of the time where regression falls clearly below that.
The ablation that isolates the key ingredient is the full analysis run without question sets, adjusting only for journal, year and topic keywords. If removing question matching leaves the vulnerability contrast and the geographic decomposition unchanged, then question matching is not doing the work we claim. A second ablation drops causal-language strength from the covariates. It tells us whether apparent "confounding rewards" are really rewards for confident abstracts.
Why not just…?
Why not just run a negative binomial regression with journal and year fixed effects? This is a real threat, and it may give the same point estimate. We will run it on identical data. The regression's weakness is that it assumes a correct parametric form while combining heterogeneous topics. The semi-synthetic benchmark will show directly whether that costs coverage under a partly observed prestige confounder. If regression with question fixed effects matches the doubly robust estimate and holds coverage, we will report the simpler model as sufficient. The contribution then rests on question matching and the exposure rubric, not the estimator.
Why not just compare retrospective findings against later randomized trials of the same question? Reversal by a trial is the cleanest evidence that an observational estimate was confounded. But such pairs are rare, and they exist mainly for high-profile questions, which would bias the sample toward heavily cited topics. We will use trial-matched questions where they exist in our corpus as a validation subset. There, high-vulnerability scores should predict a larger gap between observational and trial estimates. We do not rely on them for the main estimate.
Why not just measure spin or causal language in abstracts? Confident wording is a plausible alternative explanation, and it correlates with confounded designs. That is why we extract it separately and treat it as a mediator. If the direct effect of vulnerability disappears once causal language is included, the honest conclusion is that citing authors reward wording, not confounding. That finding would still matter.
Why not just extend venue stratification, as the geographic citation gap work did? Venue stratification cannot separate a country effect from a topic effect within a venue, and Rungta et al. state that the causal drivers remain open. The layered decomposition, from venue alone to question matching to design and effect size, directly tests how much each layer changes the gap.
Proposed Experiments
The first experiment is the semi-synthetic validation. It measures bias, root mean squared error and 95% interval coverage for the naive difference, fixed-effects regression, propensity matching, and the doubly robust estimator. It varies how strongly the unobserved prestige confounder drives both exposure and citations, and how strongly regional-journal papers drop out of tracking. The decisive check is which estimators keep coverage near nominal as confounding and dropout grow. Only those go on to the real analysis.
The second experiment is the primary within-question contrast on the WoSCC-plus-PubMed corpus. We report the vulnerability effect on five-year log citations with its interval, its mediation through effect size and causal language, and an E-value for how strong an unmeasured confounder would need to be to explain it away, as Agniel et al. did. The pre-registered falsification test uses a placebo exposure that should not affect citations: whether the paper reports a funding statement. A nonzero placebo contrast of similar size would mean the design leaves residual confounding and would invalidate the primary claim.
The third experiment is the geographic decomposition. It reports the US-versus-China and US-versus-other-country per-paper gaps after each layer of adjustment. We expect the gap to shrink under question matching but not vanish. If it instead widens, retrospective studies from lower-cited countries would be answering the same questions with equal or better designs and still be cited less. That would be the clearest evidence yet that the gap is not a quality gap.
The final check repeats the primary analysis with one-, three- and five-year windows. We also re-weight for estimated probability of remaining indexed, which tests the informative-dropout concern raised by the clinical limitations literature and applies it to our own data.
References
- Geographic Citation Gaps in NLP Research
- Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach
- Revisiting diabetes risk of olanzapine versus aripiprazole in serious mental illness care
- Target trial emulation: Do antimicrobials or gastrointestinal nutraceuticals prescribed at first presentation for acute diarrhoea cause a better clinical outcome in dogs under primary veterinary care in the UK?
- Laparoscopic versus open pancreaticoduodenectomy for pancreatic ductal adenocarcinoma: a propensity score matching analysis
- Risk factors for steroid-refractory in immune checkpoint inhibitor-induced colitis: a retrospective cohort study
- 90-day mortality prediction in elective visceral surgery using machine learning: a retrospective multicenter development, validation, and comparison study
- The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research
- Application of Artificial Intelligence in the diagnosis and treatment of colorectal cancer: a bibliometric analysis, 2004–2023
- Research progress of ferroptosis in Parkinson’s disease: a bibliometric and visual analysis
- Targeted Deep Architectures: A TMLE-Based Framework for Robust Causal Inference in Neural Networks