When doctors choose treatments based on how sick a patient looks, observational studies of those treatments inherit a bias, called confounding by indication. If every study in a literature shares the same unrecorded reason for treatment, pooling more studies does not remove the bias. It only narrows the confidence interval around the wrong answer. This proposal makes that intuition quantitative and usable. We will estimate, for a given body of evidence, how much the next study would actually reduce error, and how that value depends on the study's design. Most of the remaining information turns out to lie in studies whose prescribing practice differs from the ones already published, not in more studies of the same kind.
The concrete deliverable is a hierarchical meta-analysis model that learns the hidden treatment-selection rule from variation across studies, together with a value-of-information calculation built on it. The model handles the case where each study treats patients above a different hidden severity threshold. The calculation ranks candidate next studies: another routine cohort, a cohort at a site with different prescribing habits, a cohort that records the decision variable, or a small randomized trial. The motivating case is Bayesian Meta-Analyses Could Be More. In that work, six induction-of-labor studies pooled by standard fixed-effect meta-analysis showed mechanical dilation carrying a 39% higher C-section risk (RR 1.39, p < 0.001). Once the unrecorded cervical-readiness score (Bishop score) was modeled as a latent variable, the risk ratio fell to 1.04 (95% CI 0.93–1.18).
Core Claim
Treat each study's bias as a draw that is partly shared across studies. Let $s^2$ be the variance of study-level bias, $\rho$ the fraction of it that is common to all studies, and $v$ the average sampling variance. Then the mean squared error of a pooled estimate from $k$ studies behaves like
$$\mathrm{MSE}(k) \approx \rho s^2 + \frac{(1-\rho)s^2 + v}{k}.$$
This follows directly from averaging $k$ correlated bias terms. It implies that the marginal error reduction from study $k+1$ falls roughly as $1/k^2$ toward a floor of $\rho s^2$, while the reported interval keeps shrinking as $1/\sqrt{k}$. We call the gap between these two curves false precision. In this view, the diminishing returns described at the macro level by Science of science appear inside a single meta-analysis as a mechanical property: publication count grows, but uncorrelated information does not.
The floor is set by $\rho$, and $\rho$ is not fixed. Suppose studies differ in their hidden treatment threshold $c_j$ (the severity above which study $j$'s clinicians treat), and the latent severity distribution is shared. Then each study's treated fraction pins down $c_j$, and its arm-specific outcome rates constrain how outcome depends on severity. Cross-study spread in $c_j$ therefore identifies the bias that a common threshold would leave unidentifiable. Our central claim follows. The true measure of whether a confounded literature is saturated is how diverse its treatment-selection regimes are, not how many papers it contains. A model that exploits this diversity can replace the expert elicitation that Bayesian Meta-Analyses Could Be More required. It can also tell investigators which kind of next study is worth running.
Relation to Prior Work
Two papers supply the methods we build on, and both leave the same opening. Bayesian Meta-Analyses Could Be More models the missing Bishop score with truncated normals. Its cutoffs and hyperpriors come from structured interviews with obstetricians, it uses a single shared truncation, and it is applied post hoc to six studies. We keep its latent-decision-variable structure. We differ by letting the threshold vary by study and be learned from treated fractions, and by asking a forward-looking question that work does not ask: what is the value of the next study? Removing Hidden Confounding by Experimental Grounding corrects an observational effect model with a linear bias function fitted on a small randomized trial. It shows that extrapolating the bias is more robust than extrapolating the trial effect. We use it as the reference for what a trial buys. Our model is meant for literatures where no such trial exists, and it treats "run a small trial" as one option to be priced rather than a prerequisite.
Several papers motivate the problem by showing that standard adjustment leaves bias behind. The sepsis cohort in Prescription opioids are associated with higher mortality in patients diagnosed with sepsis kept a hazard ratio of 6.239 after adjustment for demographics and comorbidity, because acute distress was never recorded. The canine diarrhoea Target trial emulation reached a near-zero antimicrobial risk difference (0.4%, 95% CI −4.5% to 5.3%), but only by recording the clinical signs that drive prescribing. Beyond Simpson's Paradox shows a pooled −26.0 percentage-point co-authorship gap in AI-agent pull requests dissolving under repository fixed effects and workflow restrictions, a software analogue of channeling.
Is More Science Really Less Science? constrains how we frame the work. It argues that publication growth reflects real institutional expansion rather than declining quality. Our claim is consistent with that argument. We do not say papers are getting worse. We say that, under shared confounding, counting them overstates what they jointly establish.
Bias-adjusted meta-analysis has a history outside this file, and we do not claim the shared-bias floor as an original observation. The new parts are three. First, the bias is made learnable through cross-study threshold variation. Second, design-specific marginal value is computed from that learned structure. Third, those marginal-value predictions are validated against ground truth.
Research Program
The first component is the model. Each study $j$ reports, by arm, counts and outcome rates and sometimes summary covariates. Latent severity follows a shared distribution, with a study-level location shift. Treatment is assigned above threshold $c_j$, softened by a logistic "fuzziness" so that thresholds need not be exact. Outcome risk depends on severity through a shared slope and on treatment through the target effect $\theta$. Studies that record the decision variable enter as partially observed and anchor the slope. The model is fit with the same sampler family as the Bayesian meta-analysis work (NUTS in NumPyro), using weak generic priors in place of interview-elicited ones. An ablation reinstates elicited priors to measure what elicitation adds.
The second component is the marginal-value calculation. For each candidate next study, the posterior predictive simulates its data, refits, and records the expected reduction in posterior variance of $\theta$. The candidates are a routine cohort at the median observed threshold, a cohort at an extreme threshold, a cohort recording the decision variable, and a small trial of size $n$. Plotting this against $k$ gives an empirical marginal-utility curve for the literature. We will check whether it matches the $1/k^2$-to-floor shape for same-threshold additions, and whether it fails to match for threshold-diverse additions.
The third component is a literature-level diagnostic. We will compute the spread of inferred thresholds across a published corpus, and test whether low spread predicts large false precision.
Baselines
The naive reference point is the standard inverse-variance fixed-effect and random-effects meta-analysis. This is what a systematic review would report. On the induction-of-labor corpus it gives RR 1.39 (Bayesian Meta-Analyses Could Be More). Random effects matters because it is how practitioners already acknowledge between-study heterogeneity, so we report both its interval and its prediction interval.
The strongest existing method is the expert-elicited latent-variable model of Bayesian Meta-Analyses Could Be More. It reported RR 1.04 (95% CI 0.93–1.18) on the six-study, 4,037-patient corpus. We rerun it with its published structure on the same corpus and on all semi-synthetic corpora. In the synthetic settings we give it elicited priors centred on the true severity distribution, which is a generous setting. On the real corpus, beating it means reaching a comparable estimate without elicitation. In the synthetic settings, beating it means lower error and correct coverage once thresholds vary by study, the regime its single truncation cannot represent.
Where a small trial is available (the semi-synthetic Tennessee STAR setting), we run Experimental Grounding (Removing Hidden Confounding by Experimental Grounding) with the same observational data and trial subsample. Its numerical errors appear in the file only as a figure, so there is no reported numerical result found on Lacuna for STAR. We will therefore produce our own numbers under identical splits.
The ablation that isolates the key ingredient fits our model with all thresholds tied to one value. If the tied model performs as well as the full model, cross-study threshold variation is not doing the work, and the central claim fails.
All methods get the same corpus, the same sampler budget (chains, warm-up and draws matched), and the same prior-tuning allowance.
Why not just…?
Why not just report an E-value? The E-value says how strong an unmeasured confounder would have to be to explain away the result. It neither estimates the corrected effect nor ranks which study to run next. Bayesian Meta-Analyses Could Be More also argues that clinicians cannot reliably guess confounder strength in abstract units. We will still report E-values on every corpus, to show whether they flag the same literatures that our false-precision measure flags.
Why not just use random-effects meta-analysis with a prediction interval? Random effects treats study biases as independent. Its interval therefore still shrinks with $k$ when bias is shared. The sequential-accumulation experiment below tests directly whether its prediction interval covers the truth as studies accumulate. We expect it to fail when the common share of bias $\rho$ is high.
Why not just add a single shared bias term with a skeptical prior? This is a real threat. It reproduces the error floor with almost no machinery. Its weakness is that the floor's height comes entirely from the prior, so the data can never lower it, and it cannot distinguish a same-threshold next study from a threshold-diverse one. We run it as a baseline. If it matches our model on error and on marginal-value calibration, the structural model is unnecessary and we will say so.
Why not just meta-regress study estimates on each study's treated fraction? This is the simplest way to exploit threshold variation, and it may capture much of the gain. If a linear meta-regression matches the full model, the contribution narrows to the value-of-information accounting and the diversity diagnostic. Our semi-synthetic settings include nonlinear severity–outcome links where we expect linear extrapolation to break. That expectation is a hypothesis, not a result.
Why not just run a trial and use Experimental Grounding? When a trial exists, that is often the right answer, and our calculation will say so by pricing the trial option. Many confounded literatures have no trial, though, and the question of which observational study to fund next still needs an answer.
Proposed Experiments
The decisive tests use semi-synthetic literatures with known truth.
From the Tennessee STAR randomized data (Removing Hidden Confounding by Experimental Grounding), we build "literatures" of 5 to 200 synthetic studies. Each is formed by subsampling and applying a threshold-based selection on a hidden baseline covariate. We vary the across-study threshold spread from zero to wide and the severity–outcome link from linear to strongly nonlinear. Metrics are root mean squared error against the trial effect, 95% interval coverage, and false precision, the ratio of reported interval width to actual error. The decisive check is calibration of predicted marginal value. Each method predicts the error reduction from adding a specified next study, we add it, and we compare predicted to realized reduction across hundreds of replicates. A method that cannot predict when additional studies stop helping fails the test.
On the real six-study induction-of-labor corpus, we test whether our model, without elicitation, recovers an estimate consistent with the elicited result of RR 1.04. We also test whether the inferred thresholds order the studies plausibly, given the clinical account that Pitocin goes to high-Bishop-score patients.
As a large, accessible second domain, the AIDev pull-request data (Beyond Simpson's Paradox) lets us treat repositories as "studies" and per-repository naive co-authorship gaps as study estimates. The fixed-effect and multi-commit analysis of that paper serves as reference: for example, Copilot's gap fell from +36.2 pp to +4.8 pp (p = 0.59). We then run cumulative meta-analysis in a random repository order. This tests whether naive pooling converges confidently on a biased gap, and whether our model's predicted plateau matches the point where the reference analysis stops moving.
If access permits, the same exercise on clinic-level slices of VetCompass, referenced to the published target-trial-emulation risk difference, gives a clinical replication. VetCompass record access is not guaranteed, so this experiment is optional.
The proposal fails if the tied-threshold ablation or the shared-bias-prior baseline matches the full model on both error and marginal-value calibration.
References
- Bayesian Meta-Analyses Could Be More: A Case Study in Trial of Labor After a Cesarean-section Outcomes and Complications
- Science of science
- Removing Hidden Confounding by Experimental Grounding
- Prescription opioids are associated with higher mortality in patients diagnosed with sepsis: A retrospective cohort study using electronic health records
- Target trial emulation: Do antimicrobials or gastrointestinal nutraceuticals prescribed at first presentation for acute diarrhoea cause a better clinical outcome in dogs under primary veterinary care in the UK?
- Beyond Simpson’s Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship
- Is More Science Really Less Science? The Scientization of Science, 1900–2020