When we pick a causal graph from data, we usually score each candidate by how well it fits and subtract a penalty for complexity. For linear models with hidden variables or with "bows" (two variables joined both by a direct causal arrow and by a hidden common cause), the standard penalty, half the number of free parameters times $\log n$, is wrong in two separate ways. The first is that some parameters are redundant, so the model's true dimension is below the parameter count. Recent work has repaired this by computing the correct dimension. The second problem is less obvious and has not been repaired. When the true graph sits on a singular point of a larger candidate, meaning a place where the larger model's set of distributions pinches, folds, or self-intersects instead of looking locally flat, the Bayesian evidence for that candidate is controlled by a smaller, usually fractional number called the learning coefficient. Typical cases are a loading near zero, a weak instrument, or a bow edge that vanishes. These are exactly the points where parameter identifiability breaks down.
We propose to compute these learning coefficients for linear Gaussian causal graphs and use them in a singular Bayesian information criterion for structure learning. Two ideas make this tractable. The first is that the learning coefficient of an acyclic mixed graph should decompose into a sum over its confounded components. A small library of component-level coefficients, partly derived by hand and partly estimated once by sampling, could then score large graphs cheaply. The second is that identifiability analysis tells us where singularities live: the loci where the Jacobian of the covariance map drops rank are the only places that need a non-standard coefficient. Everywhere else the cheap Jacobian-rank dimension is already correct. The claim is that this matters most in the regime practitioners care about, where causal effects are weak and samples are small. That is the regime where current dimension-corrected scores should systematically miss real edges.
Core Claim
The asymptotic log evidence of a candidate graph $G$, when data come from a distribution $q$ in its model, behaves as
$$\log Z_n(G) = \log \hat L_n(G) - \lambda_G(q)\log n + (m_G(q)-1)\log\log n + O_p(1),$$
where $\hat L_n$ is the maximized likelihood, $\lambda_G(q)$ is the learning coefficient (the real log canonical threshold of Algebraic Information Geometry for Learning Machines with Singularities), and $m_G(q)$ is its multiplicity. If the model's image is smooth at $q$, then $\lambda_G(q)=\dim(G)/2$ and dimension-corrected BIC is exact to this order. If $q$ lies on a submodel where identification degenerates, then $\lambda_G(q)<\dim(G)/2$. Any score using $\dim(G)/2$ then over-penalizes the larger graph, and the score under-selects true but weak edges.
Our structural conjecture concerns acyclic mixed graphs whose districts (maximal sets of variables linked by hidden-confounder edges) have variation-independent parameters. For these graphs the Gaussian likelihood factorizes over districts. The Kullback–Leibler divergence then splits into a sum of terms in disjoint parameter blocks, and learning coefficients of such sums add:
$$\lambda_G(q)=\sum_{D}\lambda_D(q_D),$$
with $D$ ranging over districts. This additivity is the property that would let a precomputed table of district-level coefficients score graphs far larger than any symbolic method can handle.
Relation to Prior Work
Two papers motivate the gap. Score-Based Causal Discovery of Latent Variable Causal Models derives exact degrees of freedom for linear 1-factor and latent hierarchical models and plugs them into a regular BIC penalty. That fixes the first error but treats the model as regular near zero loadings, where it is not. Structure Learning for Cyclic Linear Causal Models proves that for bow-free cyclic mixed graphs the dimension equals the edge count, which justifies BIC there. It explicitly excludes bows, the case where dimension drops and the parametrization becomes singular. We take on both the bow case and the singular-submodel case.
Two papers supply the method. A Bayesian information criterion for singular models resolves the circularity that $\lambda$ depends on the unknown truth. It does this by mixing over a partially ordered set of submodels, and it shows the singular criterion finds the true rank in reduced-rank regression at $n=300$ where BIC needs far more data. It only tabulates coefficients for rank, factor, and mixture hierarchies, not for causal graphs, and that missing table is what we build. Scores for Learning Discrete Causal Graphs with Unobserved Confounders uses the widely applicable Bayesian information criterion (WBIC: the average log-likelihood under a posterior tempered at $1/\log n$) for discrete graphs with confounding. It reports that this score reaches "near-perfect identification" of the instrumental-variable graph at $n=1000$ where BIC-based scores fail. That paper is discrete-only and needs tempered sampling for every candidate graph. We work in the linear Gaussian setting and aim to replace per-candidate sampling with a reusable coefficient library. Its decomposition of the score over confounded components is the discrete analogue of our additivity conjecture.
Several papers constrain the design. Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory provides the sampling-based estimator of the local learning coefficient. It also documents that this estimator is costly and sensitive to temperature and step size, which is why we estimate each coefficient once on synthetic data with known truth rather than on every dataset. The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank gives an independent rank readout we use to check that estimator.
The identifiability literature tells us where to look. Identifying Causal Effects with Computer Algebra classifies all 4-vertex graphs by generic identifiability, and On the Complexity of Identification in Linear Structural Causal Models shows exact symbolic identification is PSPACE and numerical identification is ∀ℝ-hard. Both rule out symbolic elimination as a scoring subroutine. They also say nothing about the evidence penalty at degenerate points, which is our question. Learning on a Razor's Edge finds that singularities of polynomial networks are exactly sparse subnetworks. This is the neural analogue of our observation that singular points of mixed-graph models are sub-graphs with vanishing edges. Finally, Likelihood ratio tests and singularities and Statistical Estimation in Singular Models explain why the regular asymptotics fail there.
Research Program
The first component is theory. We will prove the district additivity of $\lambda$ for acyclic mixed graphs with variation-independent district parameters, including how multiplicities combine. We will then derive exact coefficients by hand for the motifs that recur in practice: the single bow, the instrument-plus-bow, the 1-factor model with one or more vanishing loadings, and two-level latent hierarchies with a vanishing latent-to-latent edge. The factor-analysis cases have partial results in the singular-BIC line of work, and those results serve as checks.
The second component locates singularities computationally. For each district pattern and each sub-support (a pattern of zeroed edges), we compute the Jacobian rank of the covariance map at random points of the sub-support and at random points of the full model. Where the two ranks agree and the fiber is well behaved, we set $\lambda=\dim/2$. This is cheap and covers most cases. Where they disagree, the point is a candidate singular locus. This is the identifiability-breakdown set, and it is where the third component applies.
The third component builds a library. At each flagged (pattern, sub-support) pair without a closed form, we simulate large samples from the sub-support truth and estimate the local learning coefficient with the sampler of Urdshals et al., using several temperatures and step sizes. Because true coefficients are rational with small denominators, we snap estimates to the nearest candidate rational and keep the snapped value only when the estimate is stable across settings. Every snapped entry is validated against a held-out exact derivation wherever one exists.
The fourth component is search. We embed the resulting singular criterion in the greedy searches already used by the two causal baselines. The singular criterion mixes over submodels, and the full set of sub-graphs is too large to mix over. We therefore restrict the mixture to sub-graphs within the current search neighborhood: single-edge deletions plus the zeroed-edge patterns flagged in the second component. Cyclic graphs do not factorize over districts, so they are handled only for small graphs (5 to 6 nodes) as whole-graph entries.
Baselines
The naive reference point is BIC with raw parameter count, which is what most off-the-shelf score-based learners would do on these graphs. It is included to show the size of the first error alone.
The dimension-corrected baseline replaces the parameter count with the exact model dimension. That is the degrees-of-freedom formula of Ng et al. for latent models, and the Jacobian-rank dimension for bow graphs, which extends Améndola et al. to the case they exclude. This is the strongest closed-form competitor because it already fixes everything except the singular correction. On standard 1-factor data, Ng et al. report an F1 of 0.99 at $T=100$ for their method, against 0.90 for FOFC. On bow-free 5-node cyclic graphs at $n=10^4$ from random initialization, Améndola et al. report recovering the dimension about 76% of the time and the equivalence class about 45% of the time. For bow-containing graphs, there is no reported result found on Lacuna for any method.
The strongest method-level competitor is per-candidate WBIC adapted to Gaussian graphs, in the style of Bellot et al. It targets the same quantity without a library, so it is the direct test of whether the library is accurate. It will be run on identical data and with the identical search procedure. Its tempered sampler gets the same tuning budget we spend building the library, amortized across experiments, and we report wall-clock time per scored graph alongside accuracy.
The key ablation keeps our search and mixture structure but sets every $\lambda$ to $\dim/2$. This isolates the singular coefficient as the only changed ingredient. A cross-validated held-out-likelihood score is added as a non-Bayesian reference.
Beating the dimension-corrected score means a statistically significant gain in F1 and structural Hamming distance in the weak-effect regime, with no loss in the standard regime. Beating WBIC means matching its accuracy within noise at an order of magnitude lower scoring cost. Accuracy above WBIC is not expected, since WBIC estimates the same object.
Why not just…?
Why not just use the exact model dimension in BIC, as Ng et al. do? Dimension is a first-order fix. At a singular point the evidence penalty is $\lambda\log n$ with $\lambda<\dim/2$, and Drton and Plummer's reduced-rank results show that using dimension in that situation measurably delays recovery of the true model. Whether this delay is practically large for causal graphs is unknown, and the ablation above settles it. If the singular and dimension-corrected scores tie even at small effect sizes, we will report the singular correction as practically irrelevant for these graphs.
Why not just run WBIC on every candidate graph? This is the real threat. WBIC is consistent for the same quantity and needs no library. Our case rests on cost, and on the variance of one sampling run per graph per dataset compared with a vetted, snapped table. If WBIC turns out cheap and stable enough on graphs of 10 to 20 nodes, the contribution narrows to the additivity theorem and the coefficient tables as an explanation of why WBIC works. We will state this plainly.
Why not just test edges with valid irregular-model tests, such as universal likelihood ratio tests? Gaussian Universal Likelihood Ratio Testing gives finite-sample validity without regularity. It pays for that with sample splitting, which costs power precisely in the small-$n$, weak-edge regime we target. Edge-wise testing also inherits the error-propagation problems Ng et al. document for constraint-based methods. We include a universal-test edge-pruning variant in the weak-signal experiment to measure the power gap directly.
Proposed Experiments
The first experiment checks the library before using it. On reduced-rank regression and factor-analysis hierarchies, where exact coefficients are known, snapped sampler estimates must recover the true $\lambda$. Persistent misses of more than a quarter, or snapping that is unstable across temperatures, would falsify the library approach. In that case we fall back to hand derivations for a smaller motif set. The exchange-rate data used by Drton and Plummer serves as a real-data sanity check that our criterion reproduces their factor-count weights.
The second experiment uses latent structures. We reuse the 1-factor and hierarchical generators of Ng et al. with $T$ from 100 to 10,000. We sweep the smallest loading and the smallest latent-to-latent coefficient from typical magnitudes down toward zero, so the truth approaches the singular locus at controlled rates. We report F1 and structural Hamming distance. The decisive prediction is a crossover. All Bayesian scores should tie at typical magnitudes, consistent with the reported 0.99, while the singular score should keep weak edges at smaller $T$ than the dimension-corrected score. The gap should shrink as $T$ grows, because the true model eventually leaves the singular neighborhood at the $n^{-1/2}$ resolution scale.
The third experiment uses bows and cycles. We extend the 5- and 6-node cyclic simulator of Améndola et al. by adding one or two bows, with $n$ from $10^2$ to $10^4$, and measure the same three recovery rates (dimension, skeleton, equivalence class) from random and true-graph initializations. We first replicate the bow-free 76% and 45% figures to confirm our implementation. The acyclic subset is also scored using district decomposition, which tests additivity empirically: the decomposed score must agree with whole-graph WBIC within sampling noise.
The fourth experiment uses real data. On the Sachs protein-signaling data, with intermediate proteins hidden as in Bellot et al., we compare which edges each score retains. We check whether the singular score keeps the PLCG–PIP2–PIP3 relationships that Améndola et al. recovered and whether it adds bows the bow-free method could not represent. This experiment is qualitative, as in the source papers, since no numeric ground-truth metric is reported there.
Across all experiments we report scoring time per graph. The proposal succeeds if the singular criterion beats the dimension-corrected score in the weak-effect regime, matches WBIC at a fraction of its cost, and the additivity check holds. It fails informatively if the dimension-corrected score is never beaten, which would show that for linear causal graphs the singular correction is an asymptotic curiosity.
References
- Algebraic Information Geometry for Learning Machines with Singularities
- Score-Based Causal Discovery of Latent Variable Causal Models
- Structure Learning for Cyclic Linear Causal Models
- A Bayesian information criterion for singular models
- Scores for Learning Discrete Causal Graphs with Unobserved Confounders
- Compressibility measures Complexity: Minimum Description Length meets Singular Learning Theory
- The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank
- Identifying Causal Effects with Computer Algebra
- On the Complexity of Identification in Linear Structural Causal Models
- Learning on a Razor’s Edge: Identifiability and Singularity of Polynomial Neural Networks
- Likelihood ratio tests and singularities
- Statistical Estimation in Singular Models.
- Gaussian universal likelihood ratio testing