Calibrating Propensity Score Matching Against the Ties That Bibliometric Proxies Miss

Studies in the "science of science" routinely ask causal questions. Does having an editor who is a former co-author speed up acceptance? Does a mentor's same-race pairing preference shape whether a trainee stays in academia? Did double-blind review lower scores? The usual tool is propensity score matching: estimate each unit's probability of receiving the treatment from its observed characteristics, then compare treated and untreated units with similar probabilities. The method is only as good as the confounders it sees. In science, the confounders that matter most are the relationships between people, and these are recorded through flat proxies such as shared bylines. Those proxies are known to miss informal advising, acknowledged help, mentorship without joint papers, and ties formed outside a single venue. Any "matched" comparison built on co-authorship alone is therefore balanced on a partial picture of the social graph, and nobody knows how much that matters.

This note proposes to measure that gap directly and then use the measurement. Collaboration proxies of increasing richness already exist: single-venue co-authorship, all-venue co-authorship, inferred mentorship, and acknowledgment ties. Each rung of this "proxy ladder" reveals ties the rung below missed. We re-estimate the same causal effect while balancing on each rung and track how the estimate moves. That movement becomes an empirical anchor for the one thing classical matching cannot see, the ties still missing at the top rung. In short, we turn the known incompleteness of bibliometric proxies into a calibrated sensitivity analysis for matching.

Core Claim

The claim has two parts. The first is descriptive: published matched estimates in science-of-science studies shift by a meaningful fraction when relational confounders from richer proxies are added to the balancing set. The second is methodological: the size of the shift between adjacent rungs predicts the residual bias left at the top rung well enough to produce honest intervals, meaning intervals that cover the true effect in semi-synthetic tests where the hidden ties are known.

The closest existing work handles neighboring problems. Propensity Score Matching for Causal Inference with Relational Data builds relational covariates by expanding paths up to a fixed number of hops and blocking on shared group identifiers. It assumes the graph it expands is the true graph, and it was tested on a Foursquare friendship network. Propensity score methodology in the presence of network entanglement between treatments shows that when treatment assignment depends on network formation, independent-unit propensity models are badly biased. It reports an RMSE of 57.05 versus 3.41 for its network-evolution model on 100-unit simulations, again assuming the network is fully observed. Causal Matching with Text Embeddings matches ICLR papers on SPECTER embeddings of titles and abstracts and ignores authors' ties entirely. Effects of Same-Race Mentorship Preferences on Academic Performance and Survival applies standard matching to crowdsourced mentorship pairs.

None of these papers treats the observed graph as a mismeasured version of the true one. That is the new element here. Our method borrows machinery from all four, but the question it answers is different: how wrong is the matched estimate because the proxy graph is incomplete, and can the proxy hierarchy itself bound that error?

Research Program

We build a matching pipeline whose balancing set combines three things. The first is document content, using SPECTER nearest-neighbor matching in the style of the ICLR study. The second is node-level author history. The third is relational aggregates computed from a chosen proxy graph: counts, means, and maxima over one- and two-hop neighbors, as in relational propensity matching, plus blocking on institution. The proxy graph is a swappable input.

The ladder has four rungs:

  1. Co-authorship within the focal venue.
  2. Co-authorship across all venues, drawn from the full publication record, since Quantifying the effect of editor-author relations on manuscript handling times and Sarigöl et al. note that single-venue sampling underestimates social capital.
  3. The rung-two graph plus inferred mentorship edges from S2AMP. S2AMP reports classifier AUROC 0.96 but requires at least two joint papers.
  4. The rung-three graph plus directed acknowledgment ties extracted from full text, following Giles et al..

For each rung $k$ we compute a matched estimate $\hat\tau_k$ and its covariate-balance diagnostics. The calibration step models residual bias as a function of how much new relational signal each rung adds. For each rung we measure two quantities:

  • Coverage: the fraction of ties the rung adds that the previous rung lacked.
  • Confounding strength: how strongly those new ties predict treatment and outcome.

We then extrapolate the sequence $\hat\tau_1,\dots,\hat\tau_4$ toward a hypothetical complete graph. The extrapolation is expressed in the language of standard sensitivity parameters, the strength an omitted confounder would need to have. The difference from a plain sensitivity analysis is that the plausible range of that strength is estimated from the observed rungs rather than asserted.

Demographic covariates inferred from names are handled on the same logic, as a second measurement-error axis. The mentorship dataset in A dataset of mentorship in bioscience with semantic and demographic estimations reports an F-1 of about 0.30 on African American names, against more than 0.90 for most other categories. We propagate that per-group error into matching and check how much of the same-race-mentorship effect survives.

Truth is never observed in real data, so validity is tested semi-synthetically, following the hybrid design of the relational matching paper. We take the real rung-four graph and simulate treatments and outcomes that depend on it, including acknowledgment and mentorship ties. We then hand the estimator only lower rungs. Because the simulated effect is known, we can check whether the calibrated interval covers it.

We also run a separate condition where treatment assignment is itself network-driven. In that condition the entanglement result of Toulis et al. predicts failure for every independent-unit estimator. The purpose is to verify that our calibration does not falsely certify estimates when the problem is entanglement rather than missing ties.

Baselines

Naive difference in means. This is the reference point that shows how much any adjustment matters. On the PLOS ONE editorial data the reported speedup for editor-handled prior co-authors is 19 days, and 28 days for reciprocal pairs, per Quantifying the effect of editor-author relations on manuscript handling times. On ICLR the naive score difference is −0.25 points, per Causal Matching with Text Embeddings.

Text-only embedding matching (VRM-E). This is the strongest existing matching method on scientific publication data. It reports an adjusted effect of −0.17 points on the control group and was preferred by human experts in 70% of comparisons against stratified propensity matching. We rerun it on identical samples with the same SPECTER encoder and caliper grid, adding nothing relational. Our pipeline "beats" it if, in semi-synthetic tests with hidden relational confounding, it reduces absolute bias and achieves nominal interval coverage where text-only matching does not.

Relational propensity matching on the co-authorship graph alone. This is the method of Arbour et al. using only rung one or two. It isolates whether the proxy ladder adds anything beyond relational features computed from the conventional graph. No scalar result is reported on Lacuna for that method; its Foursquare results are error curves. We tune its hop threshold over the same range and budget as ours.

Network-evolution propensity model. This is the method of Toulis et al., run in the entanglement condition. It is the right competitor when ties drive assignment. Its reported RMSE of 3.41 against 57.05 comes from 100-unit simulations, so we run it on sampled subgraphs of comparable size and report where it stops scaling.

Uncalibrated sensitivity analysis. This means Rosenbaum-style bounds or E-values, with the confounder strength left as a free parameter. No reported result was found on Lacuna for this approach on bibliometric data. We report the interval width it needs to reach coverage, compared with ours.

Ablation. The key ablation replaces the rung-derived calibration with the same extrapolation fitted to randomly permuted rung orderings. This tests whether the ordering of proxies by richness carries information or whether any added feature set would produce similar-looking movement.

Why not just…?

Why not just put network metrics into a regression? Degree or centrality entered as controls in OLS or Cox regression inherits exactly the proxy problem we study, because the metrics come from the same incomplete graph. Toulis et al. further show that independent-unit models fail under entanglement. The semi-synthetic runs will include a regression-with-network-metrics arm. If it matches our bias and coverage once given the same rung-four graph, the contribution reduces to "use richer graphs." That would still be a useful finding, but a smaller one, and we would report it as such.

Why not just use the richest proxy and stop? This is the most serious threat. If rung four already captures nearly all relevant ties, calibration adds little. The file gives reasons to expect it does not. S2AMP needs two joint papers, Ke et al. document mentors who never co-publish, and crowdsourced trees skew toward senior scientists. Still, this is an empirical question. The semi-synthetic condition where hidden ties sit entirely outside rung four answers it directly: if coverage at rung four alone is already nominal, we drop the calibration claim.

Why not just run a standard sensitivity analysis? Standard bounds are honest but usually so wide, or so dependent on an unjustified strength parameter, that they settle nothing. Our method is a way of choosing that parameter from data. If uncalibrated E-values achieve coverage with intervals no wider than ours, the calibration is unnecessary.

Why not just match exactly on co-authorship distance? Exact matching on distance categories uses only rung one or two, which is precisely the proxy the literature says underestimates ties. It appears as a special case of the co-authorship-only baseline.

Proposed Experiments

There are three real-data studies.

The first uses the PLOS ONE handling-time data (137,536 articles). The question is the effect of an editor-author co-authorship tie on time to acceptance, with acknowledgment and mentorship ties from other venues as the hidden confounders. We report how the reported 19-day figure moves up the ladder. Because the data contain accepted papers only, we flag censoring of desk rejections as a limit that no rung fixes.

The second uses the ICLR 2017–2018 single-to-double-blind transition. We ask whether adding author-network balance moves the effect beyond VRM-E's −0.17.

The third uses the linked Academic Family Tree and Microsoft Academic Graph mentorship data used by Ding. We ask how much of the reported 27.6% lower academic-survival likelihood persists after relational balancing and after demographic error propagation.

The decisive checks are all semi-synthetic, on the real rung-four graph. First, calibrated intervals built from rungs one to three should cover the simulated effect at close to their nominal rate, while every fixed-rung estimator undercovers as hidden-tie confounding strengthens. Second, the permuted-ladder ablation should lose that coverage. Third, under entangled assignment the calibration should flag failure rather than certify a biased estimate.

If the first check fails, meaning adjacent-rung movement does not predict residual bias, we still publish the descriptive ladder shifts on the three real studies. We would not claim a calibration method.