Many causal discovery algorithms, which try to recover the directed graph of cause and effect from purely observational data, look much better on simulated benchmarks than they should. Reisach et al. showed that in standard simulated data, variables further down the causal chain simply have larger variance. Sorting variables by variance, a property they called varsortability, recovers most of the causal order for free (Beware of the Simulated DAG!). A newer family of methods orders variables with the score function, the gradient of the log-density. SCORE identifies a leaf node, a variable with no children, as one whose diagonal entry of the score's Jacobian has zero variance across samples (Rolland et al.). These methods are claimed to be robust to broken assumptions (Montagna et al.), yet on the realistic industrial benchmark causalAssembly they do no better than an empty graph (causalAssembly). Nobody knows whether that failure comes from the estimator or from the data. The kernel score estimator these methods use costs cubic time and quadratic memory in the number of samples, so it has only ever been run at a few thousand samples. Its failures on hard data are therefore confounded with finite-sample noise.
This note proposes building the tool that removes that confound, and then using it to answer the question. The tool computes the kernel score-Jacobian statistics without ever storing the samples-by-samples Gram matrix (the matrix of pairwise kernel similarities between samples). It does this with fused GPU kernels, meaning a single GPU program that recomputes pairwise kernel values in on-chip memory, reduces them immediately, and never writes the matrix to main GPU memory. A matrix-free iterative solve replaces the explicit inverse. This takes the estimator from roughly $10^3$ to $10^5$–$10^6$ samples on one GPU. The scientific claim is then testable. If score-based ordering tracks real causal structure, its ordering error on internally standardized data and on causalAssembly should keep falling as the sample count grows. If it is another simulation artifact like varsortability, the error should plateau near chance. Kernel fusion is the instrument here. Varsortability bias is the question.
Core Claim
We claim two things. First, the sample-size curve of score-sortability, the agreement between the causal order and the order induced by the leaf statistic, is the right diagnostic for separating artifact from signal. Second, this curve is currently unmeasurable at the scales where the two explanations diverge. The file documents that no prior work fuses GPU kernels for sortability or score-based discovery. The closest pieces do something different. SciNO (Score-informed Neural Operator) avoids the cubic cost by swapping the kernel estimator for a learned neural-operator score model, which adds its own inductive bias. DAS (Scalable Causal Discovery with Score Matching) speeds up edge pruning, not score estimation. Our contribution keeps the exact kernel estimator the theory is stated for and makes it cheap enough to study as the sample count grows. We then use it as a measuring device on data where variance and explainability shortcuts have been removed.
We also flag a second, specific question. The theoretical leaf criterion, zero variance of the Jacobian diagonal, does not depend on units. The finite-sample comparison across candidate leaves does depend on units, because the variance of a Hessian-diagonal estimate for variable $j$ scales as the inverse fourth power of $j$'s scale. Score-based peeling may therefore partly inherit varsortability through the back door in finite samples. The large-sample tool lets us check whether that dependence vanishes as $n$ grows, as theory predicts, or persists at practical sample sizes.
Research Program
The engineering half has three parts, all aimed at the Stein Hessian-diagonal estimator used by SCORE with a Gaussian (RBF) kernel.
The first part is a tiled, fused kernel that computes the kernel-weighted sums the estimator needs, for all $d$ variables at once. These sums are the first and second derivatives of the kernel with respect to each coordinate, contracted against vectors. The kernel streams sample tiles through on-chip memory, much as FlashSchNet streams neighbor interactions (FlashSchNet), and writes out only $n \times d$ results. FlashSchNet supplies the method here: it shows that IO-aware fusion with contention-free reductions beats framework-level scatter operations.
The second part replaces the regularized inverse $(K + \eta I)^{-1}$ with conjugate gradients driven by the fused matrix–vector product. Here $K$ is the $n \times n$ Gram matrix and $\eta$ is a ridge term. This drops memory from $O(n^2)$ to $O(nd)$ and compute to $O(n^2 d)$ per solve times the number of iterations. That iteration count is controlled by $\eta$, which we sweep.
The third part handles peeling, which is where fusion fits this problem especially well. Removing variable $k$ during leaf peeling changes the RBF kernel only by dropping one term from the squared distance. The kernel is recomputed on the fly anyway, so peeling becomes a mask over coordinates with no re-materialization. The per-iteration state, the active mask and the warm-start vectors, stays resident across the $d$ peeling steps. This is in the spirit of persistent kernels that keep state on-chip across sequential steps (Kalchbrenner et al.).
Before any causal conclusion is drawn, the fused estimator must match the dense reference implementation to within a small relative error at sample sizes where the dense version fits in memory.
The scientific half uses this estimator to trace order divergence, score-sortability, and downstream SHD/SID (structural Hamming distance, the number of edge edits to the true graph, and structural intervention distance, which counts wrong interventional predictions) as functions of $n$ from $10^3$ to $10^6$. The curves are traced under three data-generation regimes that strip shortcuts to different degrees:
- raw ANM simulations (additive noise models, where each variable is a function of its parents plus independent noise), where varsortability is typically above 0.94;
- the same simulations standardized after the fact, which Reisach et al. show still leaves $R^2$-sortability intact (A Scale-Invariant Sorting Criterion);
- internally standardized SCMs, where each variable is rescaled before feeding its children, holding $R^2$-sortability near 0.5 (Standardizing Structural Causal Models).
Ormaniec et al. motivate the third regime: continuous and neural discoverers degrade badly there. Their non-identifiability proof for linear forests also constrains our interpretation. On linear forest iSCMs, a plateau at chance is the expected outcome and serves as a negative control. The informative cases are nonlinear iSCMs, where SCORE's identifiability argument still applies in principle. For each regime we also record varsortability and $R^2$-sortability, so a success can be checked against those shortcuts.
Baselines
The naive reference points are a random topological order and the empty graph. On causalAssembly, the empty graph is reportedly what current methods fail to beat; the file gives no numeric SHD or recall for it, so there is no reported result found on Lacuna for causalAssembly. Next come the trivial shortcut orderings, sortnregress and R^2-SortnRegress (Beware of the Simulated DAG!; A Scale-Invariant Sorting Criterion). They are the strongest competitors on the question of whether score ordering adds anything beyond shortcuts. Any regime in which score ordering does not beat them shows it is not extracting new information.
The strongest existing score-based method is SCORE with dense Stein estimation, paired with DAS pruning. SCORE reports SHD $47.5 \pm 4.5$ versus CAM's $54.2 \pm 5.4$ on 20-node ER4 Gaussian graphs (Rolland et al.). DAS reports 279 seconds versus over 4,100 seconds for SCORE on 200 nodes, and about 1.5 hours at 1,000 nodes (DAS). We run the dense baseline at the largest $n$ that fits on the same GPU. We then compare it with the fused estimator at matched $n$ for accuracy, and at matched wall-clock for sample efficiency. Beating it on the engineering axis means matching its outputs at small $n$ while running at 10–100 times its maximum $n$ in comparable time. Beating it on the science axis means lower order divergence on nonlinear iSCM data at the larger $n$ than dense SCORE achieves at its memory limit.
SciNO is the strongest large-sample alternative. It reports a 42.7% average reduction in order divergence on synthetic graphs and 31.5% on real data (Score-informed Neural Operator). We run it without LLM priors, on identical data and the same GPU-hour budget, with its published tuning ranges.
Two ablations isolate the key ingredients. The first is the dense estimator compiled with torch.compile at the same $n$, which isolates hand-fusion from generic compilation. The second is a subsampled estimator with the fused kernel but a fixed $n = 2000$, which isolates the effect of more samples from the effect of the new implementation.
Why not just…?
Why not just subsample to a few thousand points, as everyone does now? Whether that suffices is exactly what the proposal measures. The subsampled ablation traces the same curves. If order divergence on nonlinear iSCM and causalAssembly is already flat by $n \approx 2000$, the large-sample tool has no scientific payoff. We would report that plainly: it would mean the causalAssembly failure is not estimator-limited. That negative result is still worth having, because it would settle the tension between Montagna et al. and causalAssembly.
Why not just fuse the sortability diagnostics themselves? For the linear case this would be pointless, and we do not propose it. Varsortability is a sort of marginal variances. Linear $R^2$ for every variable follows from one precision-matrix inversion, since $R^2_j = 1 - 1/(\Sigma_{jj}(\Sigma^{-1})_{jj})$, where $\Sigma$ is the covariance matrix. That is cheap for any $d$ in the file. Candidate gap 2 in the research file dissolves under this observation, which is why fusion is aimed only at the nonlinear score estimator. There, pairwise sample interactions are the real cost.
Why not just use torch.compile or a low-rank approximation such as random features? Generic compilers may fuse the elementwise kernel evaluation. But TurboTransformers (TurboTransformers) and FlashSchNet report large gains from hand-designed reductions over framework-level fusion, and the compiled ablation tests whether that holds here. Low-rank approximations are a genuine threat on speed. However, they smooth the second derivatives, and those are precisely the quantities whose across-sample variance decides leaf selection. We include a random-feature variant at matched compute. If it matches the exact estimator's order divergence, we will recommend it over custom kernels.
Why not just use a learned score model like SciNO? Learned models scale, but their derivative smoothness comes from architecture and training. A plateau could then reflect the network rather than the data. The kernel estimator's bias is controlled by a bandwidth and a ridge term, which we sweep explicitly. That makes it the cleaner measuring device, and SciNO is run alongside it as a comparison.
Proposed Experiments
The first experiment is a correctness and scaling check. On nonlinear ANMs with $d \in \{20, 50, 200\}$, Erdős–Rényi and scale-free graphs, and Gaussian, Laplace, and Gumbel noise, we compare the fused and dense estimators on the Hessian-diagonal estimates and on the leaf chosen at each peeling step. We report maximum relative error, agreement of the chosen leaves, peak memory, and time up to the dense out-of-memory point, then continue the fused estimator alone to $n = 10^6$.
The decisive experiment traces sample-size curves of order divergence and score-sortability across three conditions: raw, post-hoc-standardized, and nonlinear iSCM data, with linear-forest iSCM as a negative control that should stay at chance. The unit-dependence question from the Core Claim gets its own check. We rescale variables by random unit factors and test whether the leaf ranking changes, and whether any change shrinks with $n$.
Real and semi-real data come last. On Sachs, varsortability is 0.57 and $R^2$-sortability 0.82 (Beware of the Simulated DAG!; A Scale-Invariant Sorting Criterion); there we report order divergence and SID against the sorting baselines. causalAssembly can generate arbitrarily many samples, which makes it the key test. A monotone improvement in SHD over the empty graph as $n$ grows would show that score ordering captures real industrial structure. A flat curve near the empty graph would show that its synthetic successes were, like varsortability, a property of the simulator.
References
- Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy to Game
- Score Matching Enables Causal Discovery of Nonlinear Additive Noise Models
- Assumption violations in causal discovery and the robustness of score matching
- extttcausalAssembly: Generating Realistic Production Data for Benchmarking Causal Discovery
- Score-informed Neural Operator for Enhancing Ordering-based Causal Discovery
- Scalable Causal Discovery with Score Matching
- $\texttt{FlashSchNet}$: Fast and Accurate Coarse-Grained Neural Network Molecular Dynamics
- Efficient Neural Audio Synthesis
- A Scale-Invariant Sorting Criterion to Find a Causal Order in Additive Noise Models
- Standardizing Structural Causal Models
- TurboTransformers: an efficient GPU serving system for transformer models