Debiased Policy Evaluation from Graph Credit Scores

Governments build roads, grids and regional integration schemes partly to draw in investment. Economists usually judge whether this worked by looking at large listed firms, which publish audited accounts. The small suppliers in those firms' supply chains are harder to see. Their financial health is usually estimated with machine-learned credit scores built from indirect traces such as lawsuits, utility bills, and links to other firms. Those traces are proxies, meaning observable stand-ins for the thing we actually care about, and they are shaped by institutions. A lawsuit count reflects how litigious the local courts are as well as how distressed the firm is. If the regions that received infrastructure also differ in court behavior, then a study that regresses "predicted distress" on "infrastructure exposure" will mix the policy effect with the proxy's regional bias. Nothing in the pipeline would flag the problem.

This note proposes a method and an open testbed for estimating how infrastructure exposure relates to small-firm distress, using cheap graph-based credit scores plus a small, randomly drawn audit sample of true outcomes. The statistical correction comes from design-based supervised learning. The contribution is to make that correction valid when the scores come from a graph neural network, a model that computes each firm's score partly from its neighbors' data. In that case the usual cross-fitting trick, which keeps each score independent of the audit labels it is checked against, silently breaks. Everything runs on a public enterprise dataset, so any reader can check the results.

Core Claim

Design-based debiasing fixes surrogate bias in downstream regressions when units are independent. Using Imperfect Surrogates for Downstream Inference (DSL) builds a corrected outcome for every unit from the surrogate and, where available, the audited label, and restores nominal coverage on text tasks. The surrogate-only approach, by contrast, covers the truth as little as 40% of the time. We claim that when the surrogate is a graph model over an enterprise knowledge graph, applying DSL naively gives intervals that are too narrow, for two reasons. First, labels leak across folds through message passing. Second, audit errors are correlated within graph communities. We further claim that fixing both problems is what makes graph surrogates usable for policy estimates. The fix has three parts: assign folds by graph community, cut cross-fold edges at inference, and use cluster-robust variance.

For each firm $i$, let $\hat f(i)$ be the graph surrogate's predicted distress, $Y_i$ the true outcome, $R_i \in \{0,1\}$ an indicator for being audited, and $\pi_i$ the known audit probability. The corrected outcome is

$$\tilde Y_i = \hat f(i) + \frac{R_i}{\pi_i}\big(Y_i - \hat f(i)\big),$$

and the regression of $\tilde Y_i$ on infrastructure exposure $Z_i$ and controls $X_i$ is unbiased only if $\hat f(i)$ was computed without access to any audited $Y_j$ in $i$'s evaluation fold. In a graph neural network, $\hat f(i)$ depends on the features and training labels of $i$'s $k$-hop neighbors. Assigning folds node by node therefore violates this requirement whenever neighbors land in different folds.

Research Program

The substrate is SMEsD, released alongside ComRisk and Bankruptcy Prediction via Mixing Intra-Risk and Spillover-Risk. It covers 3,976 Chinese small and medium enterprises from 2014 to 2021, with 11,523 civil lawsuits and a knowledge graph of shareholder and manager ties plus industry, area and stakeholder hyperedges. SMEsD has one property that makes it ideal for this work: every firm has an observed bankruptcy outcome. We can therefore hide labels, pretend only a small audit sample is observed, and compare every estimator against the full-data answer. This avoids the unverifiable closed setting of Financial Risk Analysis for SMEs with Graph-based Supply Chain Mining, whose 23.4-million-node Alipay graph no outsider can audit.

We construct two kinds of surrogate. The first is a proxy label: a litigation-based distress indicator built from recent lawsuits in which the firm is a defendant in lower-tier ("grassroots") courts. ComRisk reports that recent grassroots litigation tracks bankruptcy, while older and higher-court litigation tracks survival. The indicator is informative but institutionally shaped. The second is a model surrogate: ComRisk-style graph predictions trained on the non-audited firms. Following the logic of On the Validity of Arrest as a Proxy for Offense, where clearance rates vary by city and aggregate comparisons flip, we also build semi-synthetic variants. In these, lawsuit recording rates vary by area in a controlled way that correlates with exposure, which gives a known size of proxy bias.

Infrastructure exposure is coded from SMEsD's area field against the public phased city list for Yangtze River Delta integration. This is the policy used by Enhancing Sustainability Through Regional Integration, which found lower financing constraints (SA index 4.7% lower) and more foreign-invested firms (78.9% more) for listed firms. That paper never examined unlisted suppliers, so our estimand extends its question to the firms it could not see. If SMEsD's area field is too coarse to map to cities, exposure falls back to province-level coding or to a semi-synthetic treatment with a planted effect. The infrastructure conclusion would then be weaker, and we will report it that way.

The method has four pieces. The first is community-based fold assignment: we partition the pairwise shareholder and manager graph, for example with a balanced min-cut, and assign whole communities to folds. The second is inference-time edge cutting, so that no firm's score passes through audited neighbors in its own fold. Area and industry hyperedges are handled differently because they connect nearly everyone. They enter as covariates, and are not used as message-passing channels across folds. The third is audit sampling by community with known probabilities, so that $\pi_i$ is exact. The fourth is cluster-robust standard errors at the community level. The reused statistical machinery comes from DSL. The adaptive, multi-round audit allocation from M-estimation under Two-Phase Multiwave Sampling is the natural extension once the one-shot design works. The doubly robust estimator of Towards Efficient Inference by Incorporating Automated Computational Phenotypes supplies a covariate-shift variant, in which audit probability depends on firm features. Both of these prior works assume independent units, and that assumption is the gap we address.

Baselines

The naive reference point is surrogate-only regression. It regresses the litigation proxy, or the graph model's predictions for all firms, on exposure and controls and ignores the audit sample. It is what an applied study would actually do. DSL's surrogate-only results on text, with 40% coverage at a nominal 95% level, tell us to expect it to fail. We will measure how badly on SMEsD.

The second reference is gold-standard-only regression, which uses audited firms alone. It is unbiased by construction and sets the interval width we must beat. The DSL paper reports that correcting surrogates cuts RMSE by 14% to 31% relative to gold-only on Congressional Bills. The MPD paper reports effective sample size gains of up to 1.8x over random gold-only sampling. No reported result was found on Lacuna for either comparison on enterprise credit data.

The strongest existing competitor is standard DSL with node-level random folds and a ComRisk-style graph surrogate. It is the published method applied as-is, and it would be the default choice of a careful applied researcher. ComRisk reports an AUC of 84.83% on SMEsD, a 9.23% improvement over the best of twelve baselines. That figure bounds the surrogate quality we can hope to reproduce.

A second strong competitor is standard DSL with a tabular surrogate, a gradient-boosted tree on firm-level features. Because it uses no neighbors, node-level cross-fitting stays valid. Its weakness is predictive power: in the ST-GNN study, gradient boosting reached an AUC of 0.574 against 0.701 for the graph model on loan default.

All methods share the same audit draws, the same surrogate architecture and hyperparameter search budget, and the same controls. They are compared at audit budgets of roughly 5%, 10% and 20% of firms over many repeated draws. The ablation that isolates our key ingredient keeps the graph surrogate fixed and swaps only the fold and variance design: node folds with node variance, community folds with node variance, and community folds with cutting and cluster variance.

Beating the strongest baseline means two things. Coverage must stay within Monte Carlo error of 95% where node-level DSL falls measurably short. Intervals must also be narrower than both gold-only and tabular-surrogate DSL at equal budget.

Why not just…?

Why not just use standard DSL and ignore the graph? This is a real threat. If message passing leaks little label information on SMEsD's sparse pairwise graph, node-level DSL may already achieve nominal coverage, and our correction would add nothing. The ablation settles it directly. We will also report leakage as a function of the number of propagation layers, since leakage should grow with receptive field size. If node-level DSL covers at nominal rates at every depth we test, we will say that the graph extension is unnecessary for this data. The testbed and the finding would still stand.

Why not just add region fixed effects to the downstream regression? Fixed effects absorb proxy bias that is constant within a region. But when exposure is assigned at the region level, they also absorb the treatment in cross-section. In a panel they fail whenever court recording behavior shifts over time alongside integration, which is exactly the semi-synthetic scenario we plant. They also do nothing about neighbor-driven leakage. We include fixed-effect surrogate-only regression as an extra arm to show when it suffices.

Why not just use a tabular surrogate, which keeps cross-fitting valid for free? Tabular DSL is valid, and it is our second baseline for that reason. The question is efficiency. Graph surrogates are more accurate: ComRisk's reported gain, and the ST-GNN finding that neighbor information raised patent-data coverage from 1.5% to 22.7% of firms, both point that way. If that accuracy does not translate into narrower valid intervals after correction, the simpler route wins, and we will report that result.

Proposed Experiments

The first check verifies our implementation. We reproduce DSL's coverage behavior on the Congressional Bills setup before touching enterprise data.

On SMEsD, the decisive experiment compares estimators against the full-label answer for the exposure coefficient. We report bias, 95% interval coverage, interval width, and effective sample size relative to gold-only. This is done under three conditions: the natural litigation proxy, the planted region-dependent recording bias, and a planted exposure effect of known size. The central prediction is a gap in which node-level DSL undercovers as graph depth increases while the community-based design holds nominal coverage.

A second experiment extends to adaptive multi-round audit allocation over communities. It tests whether MPD's reported plateau after about six rounds survives under graph dependence.

A third experiment reports the substantive estimate: whether integration exposure is associated with lower bankruptcy among these unlisted firms, together with honest intervals. We compare the sign against the listed-firm financing-constraint finding.

All code, fold partitions, audit draws and semi-synthetic generators will be released, so that every number is reproducible from public data.