Zero-Shot Noise Transfer for Symbolic Regression

Symbolic regression tries to recover a readable formula, such as Bernoulli's law, from measured data. It is notoriously fragile. Small measurement errors push the search to add spurious terms, and methods that rediscover textbook physics from synthetic data with Gaussian jitter fail on real sensor streams from wind and light tunnels (Causal chambers as a real-world physical testbed for AI methodology). The usual fix is to train neural "amortized" regressors, meaning networks that read a point cloud and output an equation in one pass, on data with added noise. This only teaches them the noise they were shown. Real instruments drift, quantize, produce heavy-tailed outliers, and correlate errors over time, often in combinations no training set anticipated.

We propose to treat noise the way zero-shot learning treats unseen object classes. In zero-shot learning, a classifier recognizes a class it never saw by reading its description, a vector of attributes such as "striped, four-legged." Here the "class" is a noise process, and the description is a short attribute vector a scientist could read off a sensor datasheet or estimate from repeated measurements: drift rate, tail heaviness, autocorrelation, quantization step, outlier fraction, input-side error. This descriptor is fused with the data at the level of individual points, before the encoder aggregates them. It then steers a mixture of experts, a network whose sub-modules are switched on selectively, so that each attribute engages a specialized module while a shared module handles general equation syntax. The claim is that this factorized, description-conditioned design recovers correct equations under unseen combinations of noise attributes, including the real noise of physical testbeds. We also claim it exposes a "seen-noise bias" analogous to the seen-class bias of generalized zero-shot learning, which noise augmentation alone cannot remove.

Core Claim

Amortized symbolic regressors fail under realistic noise partly because noise is high-dimensional and compositional. A model trained on drift and on heavy tails separately has no reason to handle drift plus heavy tails. If noise is described by attributes and each attribute is handled by a separate routed expert, the model can assemble a response to a new combination. A dense model given the same description must learn every interaction jointly. The prediction is specific. On noise families seen in training, a dense model with the descriptor should match the routed model. On held-out attribute combinations, the routed model should keep a markedly higher exact-recovery rate. If this gap does not appear, the mixture of experts is unnecessary, and the contribution reduces to the descriptor-fusion protocol. The experiments are built to tell these outcomes apart.

Relation to Prior Work

The closest work conditions symbolic regression on side information about the equation, not the noise. Controllable Neural Symbolic Regression (NSRwH) adds embeddings of structural hypotheses, such as symmetries, complexity, and required operators, to a numerical embedding. It shows this anchoring keeps recovery high under 1% noise where unconditioned regressors fail. This motivates our proposal: early-fused side information regularizes against noise. We also reuse its element-wise fusion. NSRwH's hypotheses are a closed vocabulary about the target formula, and with no hypothesis it falls back to the unconditioned baseline. Our descriptor is about the measurement process, so it is available even when nothing is known about the law.

ChatSR projects set-encoder embeddings of data into a language model and follows unseen natural-language constraints such as convexity. It is the zero-shot precedent for symbolic regression, but its constraints again concern the equation, and it was evaluated on synthetic benchmarks. MMSR aligns data and equation skeletons contrastively and fuses them by cross-attention, but its authors note it assumes noise-free data. ViSymRe suppresses noisy synthesized visual tokens with biased cross-attention. The noise it handles is created by its own pipeline, not by sensors. None of these four uses expert routing, and none tests generalization to unseen noise.

Two papers supply the routing design. Decoding Knowledge Attribution in Mixture-of-Experts finds that in language-model mixtures, always-active shared experts carry general structure while routed experts refine. It also finds that shallow mixtures lose 76% of Mean Reciprocal Rank when their top experts are blocked, against 43% for deeper ones. We adopt the shared-plus-routed split and its expert-blocking analysis. The 76% versus 43% fragility result constrains us: it warns that a small routed regressor may be brittle, so we report expert-blocking results at two depths. Symbolic Mixture-of-Experts routes whole problems to frozen language models by inferred skills. That is instance-level routing from a description, which is close in spirit to routing by noise descriptor, but it never touches numerical data.

From zero-shot learning, Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly supplies the evaluation discipline. It recommends strict seen/unseen splits with no leakage and reporting the harmonic mean over seen and unseen classes, because models bias toward seen classes. It reports harmonic means often in the 20–50% range. We carry over both the protocol and the expectation that unseen performance is where methods break.

Multimodal DBN for Predicting High-Quality Answers in cQA portals constrains the fusion design. Naive concatenation into a shallow classifier did worse than the stronger modality alone. We therefore fuse the descriptor through non-linear layers inside the encoder, never by concatenation at the output.

Research Program

We would build a noise generator covering six attributes: multiplicative or heteroscedastic scale, Student-t tail heaviness, first-order autocorrelation, slow drift (linear and random-walk) on outputs or inputs, quantization, and outlier contamination. Each noise instance comes with its attribute vector. Training uses single attributes and a subset of pairs over the SRSD equation set. Held-out families are unseen pairs, unseen triples, and attribute values outside the training range, following the strict-split rule of the zero-shot evaluation paper.

Two design choices matter. First, drift and autocorrelation are invisible to the permutation-invariant set encoders used by MMSR and ChatSR. These encoders deliberately discard point order, so each point carries its acquisition time or index as an extra channel. Second, the descriptor embedding is added to every point token before the first set-attention block, so the encoder can learn to down-weight points per attribute. If it is fused only after aggregation, damaged points have already been averaged in.

The feedforward layers of the encoder become a shared expert plus several routed experts. The routing weights for instance $i$ are $g_i = \mathrm{softmax}(W d_i + U \bar{h}_i)$, where $d_i$ is the descriptor embedding, $\bar{h}_i$ is the pooled data embedding, and $W, U$ are learned. Top-$k$ experts are activated, with a standard load-balancing penalty. The decoder produces skeletons under syntax constraints, and constants are fitted by BFGS with a loss chosen from the descriptor, for example Huber loss when tails are heavy.

At test time the descriptor comes from one of three sources, all reported separately. It can be the oracle value, a value declared from sensor metadata, or a value estimated from data: from repeated measurements at fixed settings, which the causal chambers permit, or from residuals of a nonparametric fit.

We would also analyze whether routed experts align with attributes. We force or block individual experts, following the knowledge-attribution paper's procedure, and check whether blocking the "drift" expert selectively harms drift-bearing families.

Baselines

The naive reference point is an MMSR-style dense regressor trained on SRSD with only Gaussian noise and no descriptor. It represents current practice. On clean low-dimensional benchmarks, MMSR reports an average $R^2$ of 0.9934 (MMSR). A large drop under our held-out noise is the expected failure, and it calibrates how much room exists.

The strongest neural competitor is a dense model with the same architecture, the same full noise randomization, and the same per-point descriptor fusion, NSRwH-style but without routing. It has a parameter count matched to the routed model's active parameters, the same data, the same training steps, and the same hyperparameter search budget. It is the strongest competitor because it has every ingredient except factorization. Beating it means a statistically significant gain in exact solution rate on unseen attribute combinations, with seeds and equations as the units of variation, while staying within noise of it on seen families. The pattern matters more than a uniform improvement.

A second ablation removes the descriptor but keeps routing on data alone. This isolates whether the description, rather than extra modules, drives the result.

The strongest classical competitors are genetic programming systems. Contemporary Symbolic Regression Methods and their Relative Performance reports that AI Feynman reaches a 53% exact solution rate at minimal noise. It also reports that above noise level 0.01, DSR, gplearn, AFP_FE, and AFP recover exact solutions most often, while Operon, the black-box accuracy leader, struggles at exact recovery. SRSD reports DSR falling to 43.3% on SRSD Easy and 8.00% on SRSD Hard. These figures are reference points for clean or simply-noised settings, not for our noise families. We will run DSR, AFP_FE, and PySR ourselves on identical noisy datasets. Each gets a fixed per-dataset wall-clock budget and a robust loss chosen from the same descriptor, so classical methods get the same information.

For language-conditioned zero-shot behavior, ChatSR reports a 98.5% success rate at generating convex functions when prompted. We will re-implement its set-encoder-plus-projection design on a small open language model and give it the noise description as text. This tests whether free-text conditioning already suffices. For the physical testbed, we found no reported result on Lacuna for symbolic regression numbers on the causal chambers. That paper states only qualitatively that methods struggled.

Why not just…?

Why not just train a dense model on every noise type you can simulate? This is the most serious threat. If the simulated families span the test noise, plain randomization may suffice. Our held-out-combination split is designed to remove that coverage, and the causal chambers supply noise nobody simulated. If the dense-plus-randomization model matches ours on unseen triples and on chamber data, we will report that routing is unnecessary and that the descriptor protocol is the finding.

Why not just denoise the data first and run an ordinary regressor? Smoothing removes high-frequency noise but also removes genuine periodic structure. SRBench++ already identifies periodic components as where extrapolation breaks (SRBench++). Smoothing also does nothing about drift or input-side error. We include a smoothing-plus-MMSR pipeline and measure exact recovery and extrapolation $R^2$ separately on periodic and non-periodic equations.

Why not just use genetic programming with a robust loss? Under higher noise, some genetic programming methods are the strongest existing recoverers (Contemporary Symbolic Regression Methods and their Relative Performance). We therefore give them the descriptor-chosen loss. Their weakness is per-dataset search cost and bloat, not missing information. If they win on chamber data at equal wall-clock time, the amortized approach loses on its own terms, and we will say so.

Why not just prompt a ChatSR-style model with the noise description in words? This tests whether a language interface already gives compositional zero-shot transfer. ChatSR's success concerned properties of the output equation, which the language model can check against its own output. Noise properties must instead change how the input points are weighted. The re-implemented baseline will show whether text conditioning reaches that weighting.

Proposed Experiments

All runs use SRSD Easy, Medium, and Hard with injected noise families, the Feynman set for comparability with prior noise sweeps, and the SRBench ground-truth track noise levels. The headline metric is exact symbolic solution rate, checked by SymPy equivalence, with $R^2$, node count, and SRBench++-style extrapolation $R^2$ reported alongside.

Following generalized zero-shot practice, we report the harmonic mean $H = 2 S_s S_u / (S_s + S_u)$, where $S_s$ and $S_u$ are solution rates on seen and unseen noise families. A model can then not hide unseen failure behind seen-family accuracy.

The first decisive check is the compositional gap. The difference between the routed and dense-with-descriptor models should be near zero on seen families and clearly positive on unseen triples.

The second is transfer to the causal chambers' wind and light tunnels, with Bernoulli's principle and Malus's law as ground truth. Exact recovery is judged with oracle-free descriptors estimated from repeated settings. Success here, against the documented failure of methods tuned on synthetic data, would be the result that matters most to practitioners.

The third check is descriptor dependence. Moving from oracle to declared to estimated to absent descriptors should degrade performance gracefully. A collapse at "estimated" would mean the method is only usable when noise is already characterized.

The fourth is the expert-blocking study at two depths. Attribute-selective damage would show that routing learned the intended factorization. Uniform damage would show it did not, regardless of headline accuracy.