When we only observe data, and cannot intervene, many causal arrows cannot be told apart. The standard escape is the additive noise model: if $Y = f(X) + \epsilon$ with noise $\epsilon$ independent of $X$, then regressing $Y$ on $X$ leaves residuals independent of $X$. Regressing the other way almost always leaves residuals that still depend on $Y$. This test only works if the regression recovers something close to the true $f$. Current methods get $f$ either from a fixed smoother, such as a Gaussian process, or from a fixed dictionary of basis functions, such as truncated Legendre polynomials. When the true mechanism contains a fractional power, an exponential decay, or a composition that the dictionary cannot express, the forward residuals keep structure too. The asymmetry that should orient the edge disappears.
We propose to fit each candidate direction by searching a learned continuous space of symbolic equations. The search target is residual independence rather than curve fit. The space comes from an encoder–decoder trained on millions of synthetic formulas, so every point decodes to an equation. For every edge that observational data leaves undirected, we give the forward and backward direction the same search budget to find an equation whose residuals are independent of its inputs. We orient the edge toward the direction that succeeds. Because the space covers transcendental and composite forms, the fixed-dictionary bias should shrink. Because the search runs only on undirected edges, starting from a data-conditioned initial guess, the cost grows with the number of such edges rather than with the number of candidate graphs.
Core Claim
Fixed-basis additive-noise orientation has a specific, testable failure. When the mechanism lies outside the basis, the misspecification bias in the residuals does not shrink with more data. As sample size grows, the independence test therefore rejects both directions, and orientation accuracy plateaus or falls. We claim that searching a generative equation space with an independence objective removes this plateau for mechanisms that are compact formulas. We also claim that the same search yields a second, weaker signal for free: a compact forward law often has an inverse with no compact closed form. The best backward equation's length then adds evidence about direction. That second signal is not guaranteed (the inverse of $\exp$ is $\log$), so we treat it as an ablation, not a pillar.
The closest work in the file sits on pairs of these directions and never joins all three.
GenSR builds the continuous equation space we reuse. It has a two-branch conditional VAE trained on 5 million equations: one branch maps data directly to a latent point, the other encodes equations. A modified evolution strategy (CMA-ES, which adapts a search distribution over continuous points) then refines the latent point. GenSR optimizes fit ($R^2$) and says nothing about direction.
Grammar Variational Autoencoder and DIFER likewise optimize scalar fit or downstream accuracy in a latent space over parse trees.
On the causal side, Hoyer et al. and Mooij et al. supply the identifiability principle and the benchmark. They use fixed nonparametric regressors.
Ramsey & Andrews' truncated basis function scores scale nonlinear discovery to 100 nodes and 100,000 samples. They do this with a fixed Legendre expansion and acknowledge that it degrades under non-additive and non-polynomial mechanisms.
MetaPhysiCa and invariant function learning join symbolic structure with causal invariance, but they need multiple environments. Our setting is a single observational dataset.
What is new is the objective and the placement. We run latent equation search with a residual-independence objective, symmetrically in both directions, and only on the edges left undirected by a fast skeleton method.
Research Program
The pipeline has three stages.
First, a fast constraint-based pass finds the skeleton and the edges it cannot orient. We use PC-Max with the basis-function likelihood-ratio test of Ramsey & Andrews. Its polynomial approximation is adequate for detecting whether two variables are linked at all, even where it is too crude to orient the link.
Second, for each undirected edge $X - Y$ with already-oriented parent set $P$, we search twice. In the forward search, $X$ joins $Y$'s inputs, and we solve
$$\min_{z}\; \mathrm{HSIC}\big(Y - f_z(X, P),\; (X, P)\big) + \lambda\, |f_z|.$$
Here $z$ is a point in the equation latent space and $f_z$ is its decoded equation, with numeric constants refit by least squares. HSIC is the Hilbert–Schmidt independence criterion, a kernel measure of statistical dependence. $|f_z|$ is the decoded expression's node count, and $\lambda$ is a small complexity weight. The backward search is symmetric, with $Y$ joining $X$'s inputs. Both searches start from the data-conditioned prior-branch guess and run GenSR's diagonal CMA-ES for the same number of evaluations.
Third, we orient the edge by comparing the two directions' best independence-test p-values. We abstain when both reject. We then propagate orientations with the usual orientation rules so that no cycles or new colliders appear.
Two engineering questions will be answered empirically rather than assumed. The first is whether an equation space trained for fit places residual-independent solutions near its prior-branch guess. If not, we fine-tune the prior branch on synthetic pairs, using independence-optimal equations as targets. The second is how often small latent moves decode to the same equation. DIFER documents this plateau problem. We will measure the fraction of wasted evaluations and cache repeated decodes.
Baselines
The naive reference is a coin flip on each undirected edge, alongside the unoriented output of the first stage. Any method must beat 50% arrow accuracy on those edges and must not lower the precision of edges already oriented by colliders.
On bivariate pairs, the strongest established method is Gaussian-process regression with a permutation HSIC test (ANM-pHSIC), as benchmarked by Mooij et al.. It is the incumbent because it is basis-free in principle and needs no symbolic machinery. On the real CauseEffectPairs benchmark, that paper reports 63% ± 10% accuracy and 0.74 ± 0.05 AUC for ANM-pHSIC. Beating it means a higher weighted accuracy, with intervals that do not overlap, under the same pair weighting and the same subsampling.
On graphs, the strongest competitor is BF-BIC with Best-Order Score Search from Ramsey & Andrews. It is the method whose fixed basis we claim is the bottleneck, and it is already fast at 100 nodes. The file states only that it achieved "consistently higher" adjacency and arrowhead precision on 100-node networks. We found no reported arrowhead-precision value on Lacuna for BF-BIC/BOSS, so we will rerun it on identical data. We will sweep its truncation order over the same wall-clock budget our method uses. Beating it means higher arrowhead precision on the edges in dispute at equal adjacency precision.
We add a flexible neural-network regression plus HSIC baseline, trained with the same independence objective. It is the most direct basis-free competitor. We also add an equal-budget discrete symbolic search: genetic programming with HSIC as its fitness. For CausalTab (CausalTab) we run the observational-only regime, since its reported F1 of 0.952 is for mixed interventional data.
Two ablations isolate the key ingredients. The first runs the same latent search with a fit objective, followed by a post-hoc HSIC test. It asks whether the objective matters or only the expressive space. The second keeps the independence objective but restricts the "space" to a Legendre basis of matched parameter count. It asks whether the equation space matters or only the objective. A third ablation drops the complexity-asymmetry signal.
Why not just…?
Why not just raise the Legendre truncation order in BF-BIC? Higher orders approximate $x^{1.5}$ or $e^{-x}$ better inside the data range, but they still leave residual structure near the edges of the range. They also add terms that fit noise, which is the dictionary-growth problem described by Makke. Our sweep over truncation order measures this directly. If some order closes the gap at equal cost on non-polynomial mechanisms, the claim that basis dependency is the bottleneck for orientation is wrong, and we would report that.
Why not just use a neural network or Gaussian process with HSIC? This is the real threat, because a sufficiently flexible regressor has no basis to be missing. Our bet is narrower. A compact symbolic prior should reach independent residuals with fewer samples and with less risk of the backward fit "absorbing" dependence by overfitting. That risk grows as regressors get more flexible. The matched neural-network baseline, run across sample sizes from 100 to 10,000, settles it. If it matches us everywhere, the proposal reduces to interpretability, and we would say so. We also expect to lose on the neural-mechanism graphs, where the symbolic prior is wrong by construction.
Why not just run PySR-style discrete search in both directions with an HSIC fitness? It is conceptually the same procedure. GenSR reports that continuous search solved Strogatz systems in about 33 seconds, against 1,135 seconds for FEAT and over 17,000 for SBP-GP. It also held $R^2 > 0.97$ under 10% noise. That speed matters because we search twice per disputed edge. We compare at equal evaluation budgets. If discrete search matches accuracy at similar wall-clock time, the latent space adds nothing.
Why not just use an amortized causal foundation model? CausalTab is fast. Without interventional masks, however, it struggles to resolve direction, which is exactly the regime here. It also returns no mechanism that can be checked.
Proposed Experiments
We start with controlled bivariate pairs. Mechanism families are polynomial (where fixed bases should be fine), transcendental, fractional-power, and compositional. We also include inverse-simple pairs such as $\exp$ and $\log$, where the complexity signal should vanish. Noise is uniform, Laplace, or bimodal, and sample size runs from $10^2$ to $10^4$. The decisive check is the shape of accuracy against sample size. For non-polynomial mechanisms, the fixed-basis baseline should show a rising both-directions-rejected rate as $n$ grows, while our method's accuracy should rise monotonically. If both curves rise together, the misspecification story is false.
We then evaluate on CauseEffectPairs against the 63% ± 10% ANM-pHSIC figure. We also run the Old Faithful, Abalone and altitude–temperature pairs of Hoyer et al., checking that our forward p-values are at least as large as the reported 0.5, 0.19 and 0.017, with backward p-values still near zero. We also report the recovered equations.
For graphs, we generate 20–100 node DAGs whose mechanisms are drawn from Feynman-benchmark formulas. We score arrowhead precision on disputed edges and runtime against BF-BIC/BOSS. We repeat on the MLP-mechanism neural causal model simulations of Ramsey & Andrews as the adversarial case for a symbolic prior. The success criterion is a clear arrowhead-precision gain on symbolic-mechanism graphs, no significant loss on MLP-mechanism graphs, and runtime within a small constant factor of BF-BIC on the 100-node setting.
References
- GenSR: Symbolic regression based on equation generative space
- Grammar Variational Autoencoder
- DIFER: Differentiable Automated Feature Engineering
- Nonlinear causal discovery with additive noise models
- Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks
- Scalable Causal Discovery from Recursive Nonlinear Data via Truncated Basis Function Scores and Tests
- MetaPhysiCa: Improving OOD Robustness in Physics-informed Machine Learning
- Discovering Physics Laws of Dynamical Systems via Invariant Function Learning
- CausalTab: Pretraining Across Causal Environments for Tabular Causal Discovery
- Data-driven discovery of Tsallis-like distribution using symbolic regression in high-energy physics