Catching Recycled Cohorts Before an AI System Pools Them

Automated meta-analysis systems now search, screen, extract and pool hundreds of papers in minutes. A meta-analysis combines effect estimates from many studies into one more precise estimate. Every one of these systems silently assumes that each included paper reports a different group of people. In practice, the same trial or patient registry is often published several times: a main report, a secondary analysis, a subgroup paper, a long-term follow-up. When a pipeline pools those papers as independent, it is not just making an error. It is making a confident, fluent, numerically precise claim of independence that nothing in the source supports. That is a form of hallucination that current hallucination detectors never look for, and its result is a confidence interval that is too narrow.

We propose to build and test a cohort-overlap auditor that sits between data extraction and pooling. A language model reads each candidate pair of papers and proposes whether and how their participants overlap. A deterministic checker then tests whether the proposal is arithmetically and logically possible given the reported numbers, such as sample sizes, baseline tables, recruitment windows, sites and registry identifiers. Only the pairs the checker cannot settle go to a human reviewer, who sees the specific constraint in question rather than two full papers. The benchmark that trains and evaluates the auditor is built the same way, with models drafting, programs fixing the ground truth, and humans verifying.

Core Claim

Detecting duplicated cohorts across clinical primary studies can be split into two parts. The first is evidence extraction, which language models do adequately. The second is a feasibility judgment, which arithmetic does better than any model. Consider a subset claim: "paper B's patients are a subset of paper A's." It implies hard constraints. B's count in every baseline category must be at most A's, B's recruitment window must lie inside A's, B's sites must be a subset of A's, and B's reported means must fall within the range that a size-$n_B$ sub-sample of A's distribution could produce. A model that asserts overlap between two unrelated trials with similar demographics, or asserts independence between a parent trial and its secondary analysis, will usually violate one of these constraints or fail to cite one. We claim this extract-then-check design lowers the false-merge rate well below that of a model judging the pair directly. It also lowers the missed-duplicate rate below that of registry-ID matching on reports that lack IDs. And it cuts human adjudication to a small, focused fraction of pairs.

The stakes follow from simple statistics. Suppose two estimates have the same variance $\sigma^2$ and are correlated with coefficient $\rho$ because they share participants. Their average then has variance $\sigma^2(1+\rho)/2$, while a pipeline assuming independence reports $\sigma^2/2$. For an exact duplicate, where $\rho = 1$, the reported variance is half the true one. Shinozuka et al. document this inflation of power and narrowing of intervals. The correction to the psychedelics meta-analyses shows that it changes published conclusions: removing three papers whose data duplicated other included studies shifted the 11-dimension altered-states results.

Where This Sits

The meta-analysis literature motivates the problem but supplies tools for other settings. GEAR detects shared individuals across genome-wide association cohorts from summary statistics alone. One of its methods, pseudo profile score regression, recovered 2,934 shared controls in the Wellcome Trust Case Control Consortium data with no false positives. This is the closest precedent for our checker, because it infers overlap from what was reported rather than from raw data. However, it needs allele-frequency fingerprints and cooperation between consortia, and neither exists for clinical trial reports. Lakiotaki et al. flag duplicated omics datasets, where data files can be compared directly. The citation-matrix and Corrected Covered Area approach (He et al., applied in umbrella reviews such as Pan et al., Feng et al. and the tranexamic acid umbrella review) measures how much several meta-analyses reuse the same primary studies. It assumes the identity of each primary study is already known, and our problem sits one level below that. Kondô et al. and Chen et al. show that hidden overlap is routine in registry-based reviews, which is why registry IDs alone will not be enough.

The automation literature sets the target and its blind spot. Manalyzer runs search, screening, extraction, checking and pooling end to end. It requires extracted numbers to cite their location in the source, but it treats every accepted paper as an independent observation and includes no human review. SurveyMaster and GLIMPSE treat documents as distinct sources. GLIMPSE in particular rewards content that distinguishes one document from another, which is a different notion of redundancy from shared participants. Our auditor is a module that any of these pipelines could call before pooling.

The hallucination literature supplies both methods and constraints. EG-VAR shows that letting an untrusted model propose claims while a formal kernel verifies them against tool outputs keeps numerical claims faithful to their sources. It reports 100% source-faithfulness under counterfactual perturbations, against 80–90% for a same-tool baseline. It works on single tables, and our checker brings the same "propose, then verify, else abstain" logic to claims that span two documents. CiteGuard supplies the routing design. It scores cheaply, calibrates the scores into probabilities, and sends only ambiguous cases (18% of citations) to a model. We route ambiguous pairs to humans instead. Tang et al. act as a constraint: giving the model longer context made its medical summaries less factual. This is a direct warning against the obvious design of pasting both full papers into a prompt. HDI shows that a model can produce correct-looking outputs from memorized public data rather than the supplied record. For well-known trials, a model may "know" two reports share a trial without reading them, so we test separately on obscure reports. From Detection to Diagnosis synthesizes perturbed training data with localized error spans, but only for single Wikipedia-based statements. We carry its perturbation idea to the level of cohorts.

Research Program

The first piece is a benchmark of paper pairs labeled with how their participants relate: independent, identical cohort, subset (secondary analysis, subgroup or follow-up), or partial overlap (shared sites or overlapping recruitment). Each label carries the evidence fields that justify it. Real positives come from three sources: umbrella reviews whose published citation matrices and appendices note multiple reports of the same trial, published corrections such as the psychedelics case, and Cochrane reviews, which group multiple reports under a single study. Real hard negatives are pairs of different trials matched on condition, intervention, country and era, because these are exactly the pairs where a model is tempted to merge.

To reach scale, we add human-in-the-loop synthetic pairs grounded in the ClinicalTrials.gov reports underlying NLI4CT. A program draws a true subgroup or follow-up from a parent trial's reported structure and computes every number in the derived "paper," following the ToolQA practice of fixing gold answers by execution. A model then writes the prose, and a human checks only that the prose matches the computed numbers. Bae et al. report that this kind of model-drafts, human-filters review is about 13 times more efficient than writing from scratch.

The second piece is the auditor itself. A blocking step proposes candidate pairs from shared authors, sites, interventions and dates. An extraction model then fills a fixed schema for each paper: registry IDs, recruitment window, sites, sample size, and baseline categorical counts and means with their spread. Each field must cite its source location, as Manalyzer requires. The model then proposes an overlap type. A deterministic checker tests the constraints implied by that type, including count bounds, nested date ranges, nested site sets, and whether a subset mean is attainable given the parent's mean, spread and the subset size. The result is a verdict of consistent, violated, or undetermined because the needed fields are missing. Calibrated scores decide which pairs are accepted automatically and which go to a human. The human sees the proposed relation, the one or two constraints that decide it, and the cited source spans. This is our answer to the verification fatigue identified in the human-in-the-loop literature.

The third piece is the downstream test. We run a Manalyzer-style pooling pipeline with and without the auditor, and compare its pooled estimates and intervals to gold meta-analyses deduplicated by hand.

Baselines

The naive reference is no deduplication: every included paper is pooled as independent, which is what current end-to-end systems do. This sets the downstream gap the auditor must close. On the detection task, its recall is zero by construction.

The first real baseline is registry-ID and title matching. It flags a pair when the papers share a ClinicalTrials.gov or other registry number, or when their titles and author lists are highly similar. It is cheap, transparent and perfectly precise when IDs exist, which is why it is the competitor a meta-analyst would actually use. We run it on the identical benchmark and report results separately for pairs where both papers carry an ID and pairs where at least one does not. To beat it, the auditor must match its precision on ID-bearing pairs and achieve substantially higher recall on ID-free pairs at a false-merge rate no worse than five percentage points above it.

The strongest existing method is a frontier model judging the pair directly. We give it both papers' methods and baseline sections, a structured prompt, and the same location-citation requirement Manalyzer uses. We also add Manalyzer's independent checker loop, which re-queries the model when its answer is inconsistent. This is the strongest available competitor because Manalyzer's extraction and checking design is the best published automated meta-analysis pipeline in the file. On its own benchmark it reports a screening F1 of 76.8 against 63.2 for GPT-4o, and a Level 2 Agriculture extraction hit rate of 64.37% against 22.12% for GPT-4o (Manalyzer). Those numbers concern screening and extraction, not overlap. We found no reported result on Lacuna for cross-paper cohort-overlap detection by any method. The direct-judgment baseline gets the same base model, the same input text and the same number of model calls as the auditor, and prompt tuning for both is done only on the development split. Beating it means higher macro-F1 across the four overlap types and, more importantly, a lower false-merge rate on the matched hard negatives, with confidence intervals that do not overlap.

The ablation that isolates our key ingredient keeps the auditor identical but removes the feasibility checker. The model's extracted fields are shown back to it, and it decides alone. If this ablation matches the full system, the claim that arithmetic verification is what matters is false. A second ablation removes the human route and accepts every automatic verdict, which measures what the human step buys per minute of reviewer time.

Why not just…?

Why not just match registry IDs? This is the most serious threat, and we treat it as one. If most real overlapping pairs share a registry number, ID matching solves the bulk of the problem at almost no cost. Our contribution then shrinks to the residual: registry-based observational studies, older trials, and secondary analyses that omit the parent ID. Kondô et al. and Chen et al. indicate that this residual exists in registry-based reviews, but they do not tell us its size. The first experiment therefore measures ID coverage among the real positives. If ID matching already reaches high recall, we will reframe the work as auditing ID-free observational literature rather than claim a general win.

Why not just ask a strong model whether two papers share patients? The file gives two concrete reasons to expect this to fail in specific ways. Tang et al. found that more source text made medical summaries less factually consistent. HDI showed that models can produce correct-looking outputs from memorized public cohorts rather than the supplied record, so a model may answer correctly on famous trials and guess on obscure ones. The direct-judgment baseline and the checker ablation test this directly. We also split results by how often the trial is cited, to separate reading from recall.

Why not just compute a citation matrix and Corrected Covered Area? That method, as described by He et al., measures how much several reviews reuse the same primary studies. It needs the identity of each primary study as input, so it cannot tell when two differently titled papers report one cohort. It is complementary to the auditor, not a substitute, and the auditor's output could feed it.

Why not just fine-tune a detector on synthetic pairs? Following From Detection to Diagnosis, one could train a small model on synthetic positives and negatives and skip the checker. We run this as an additional baseline on the same synthetic training data. We expect it to fit the synthetic generator's regularities and to transfer poorly to real hard negatives, but that expectation has to be confirmed. If it transfers well, the checker's remaining value is interpretability for human reviewers, and we would report that as a smaller claim.

Proposed Experiments

The detection experiment evaluates every method on the real portion of the benchmark, split by overlap type, presence of an ID, and citation frequency. It reports macro-F1, the false-merge rate on matched negatives, and the missed-duplicate rate. The synthetic portion is used for training and for stress tests in which we remove fields one at a time, to see when the checker correctly abstains instead of guessing.

The human-loop experiment gives clinician-trained reviewers the same escalated pairs in one of two formats: full papers, or the proposed relation with its deciding constraint and cited spans. We measure accuracy, time per pair, and accuracy drift over a long session. The goal is to test whether focused evidence counters the automation bias identified in the human-in-the-loop literature, using the same within-reviewer design that CiteGuard's 72% reviewer-preference study suggests is feasible.

The decisive downstream experiment rebuilds several published meta-analyses whose duplicates are known, starting with the corrected psychedelics analyses and umbrella reviews with published citation matrices. It runs them through the pooling pipeline under four conditions: no deduplication, ID matching, direct model judgment, and the auditor. Each condition is compared to the hand-deduplicated gold standard on pooled effect error and on whether the reported 95% interval covers the gold estimate. The proposal succeeds if the auditor recovers the gold conclusions where the other conditions do not, while sending under a fifth of pairs to humans.

References

  1. Correction: Synergistic, multi-level understanding of psychedelics: three systematic reviews and meta-analyses of their pharmacology, neuroimaging and phenomenology
  2. Across-cohort QC analyses of GWAS summary statistics from complex traits
  3. BioDataome: a collection of uniformly preprocessed and automatically annotated datasets for data-driven biology
  4. Aldehyde dehydrogenase 2 rs671 polymorphism and multiple diseases: protocol for a quantitative umbrella review of meta-analyses
  5. Does Tranexamic Acid Reduce the Blood Loss in Various Surgeries? An Umbrella Review of State-of-the-Art Meta-Analysis
  6. Comparative efficacy and safety of neoadjuvant immunotherapy vs chemotherapy in resectable head and neck squamous cell carcinoma: an umbrella review of randomized controlled trials and single-arm studies
  7. Use of a Pulmonary Artery Catheter in Patients With Cardiogenic Shock ― A Systematic Review and Meta-Analysis ―
  8. Risk of adverse outcomes in inflammatory bowel disease patients infected with SARS-CoV-2: a systematic review and meta-analysis
  9. Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System
  10. Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis
  11. GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews
  12. Evidence-Grounded Verified Agentic Reasoning: A Path Toward Eliminating LLM Hallucination in Empirical Inference via Tool-Attested Kernel Proofs
  13. CiteGuard: Retrieval-Augmented Citation Verification for LLM-Powered Peer Review
  14. Evaluating large language models on medical evidence summarization
  15. The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
  16. From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis
  17. NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
  18. ToolQA: A Dataset for LLM Question Answering with External Tools
  19. Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models