Overcoming Noise Sensitivity in Symbolic Discovery

47 concepts40 papers2 proposals

When fitting a curve to data points with a bit of measurement jitter, a standard deep neural network behaves like a flexible sheet of rubber. It bends slightly to accommodate the points, but its continuous nature and optimization regularizers keep its overall shape smooth.

A symbolic discovery algorithm, by contrast, behaves like a collection of rigid Lego bricks. To perfectly hit every jittery, noisy point, the algorithm cannot simply "bend" its parameters. Instead, it must alter its discrete structure—perhaps adding a highly complex combination of operators, like an extra high-frequency sine wave or a high-degree polynomial term.

This structural vulnerability is known as noise sensitivity. While continuous models smooth over irregularities, symbolic models frequently mistake random fluctuations for meaningful physical laws. Understanding why this happens—and how modern architectures circumvent it—is central to deploying AI for true scientific discovery.

The Anatomy of Symbolic Overfitting

In traditional machine learning, overfitting manifests as high variance in a model's continuous weight space. In symbolic discovery and symbolic regression (SR), overfitting manifests as structural mutation. Because symbolic systems search through an infinite, discrete space of mathematical operators, constants, and variables, even a tiny amount of stochastic noise can radically alter the optimal expression graph.

As noted by de França et al. in their comprehensive evaluation of modern tools via SRBench++, stochastic noise in target variables frequently forces algorithms to incorporate spurious features and complex mathematical terms to minimize numerical residuals. This behavior destroys the model's interpretability and masks the true underlying signal.

This brittleness stems from the fact that symbolic expressions lack the natural smoothing properties of continuous functions. d'Ascoli et al. point out in Deep symbolic regression for recurrence prediction that because symbolic regression seeks an exact mathematical expression, minor perturbations in the input sequence can cause the generator to select entirely incorrect operators or constants.

Clean Data:  y = 2x  ----> Simple, interpretable graph
Noisy Data:  y = 2x + 0.01*sin(100x) ----> Chaotic, overfitted graph

This structural fragility is not unique to algebraic equations; it spans logical induction as well. For instance, Camacho et al. observe in Learning Interpretable Models Expressed in Linear Temporal Logic that exact symbolic learning methods fail when encountering mislabeled data or sensor errors because they attempt to find a perfect logical fit for every observation. Similarly, Ardon et al. show that a single mislabeled event can derail finite-state controller induction in multi-agent tasks, leading to highly convoluted or entirely incorrect logical structures.

The Simulation-to-Reality Gap

Historically, automated discovery systems were validated primarily on clean, idealized synthetic datasets. Early landmark systems like BACON.5, developed by Langley et al., proved that general-purpose heuristics could rediscover classic physical laws (BACON.5: The Discovery of Conservation Laws), but they struggled when noise levels exceeded simple tolerance thresholds. Jones argued that symbolic discovery systems requiring absolute regularity are fundamentally vulnerable, as a single anomalous data point can cause the premature rejection of a broadly accurate hypothesis (Generating Predictions to Aid the Scientific Discovery Process).

A large-scale benchmark by La Cava et al. (Contemporary Symbolic Regression Methods and their Relative Performance) confirmed that while many modern symbolic regression methods excel in low-noise environments, their ability to recover exact solutions degrades sharply under higher noise conditions.

This gap becomes a major bottleneck when moving from mathematical simulations to physical hardware. In a study introducing real-world physical testbeds, the authors of Causal chambers as a real-world physical testbed for AI methodology demonstrated that algorithms which successfully recover natural laws from synthetic data frequently fail when confronted with real-world sensor data. Real physical systems introduce non-Gaussian noise, measurement jitters, and calibration drift—complexities that standard symbolic search heuristics are unequipped to filter out.

Mechanisms for Noise Robustness

To bridge this gap, researchers have developed frameworks that prevent symbolic search algorithms from chasing noise. These methods broadly fall into three categories: physical grounding, robust structural optimization, and learned probabilistic priors.

1. Grounding the Search Space with Physical Invariants

The most direct way to prevent an algorithm from inventing spurious terms to fit noise is to restrict the search space to equations that are physically plausible. By enforcing domain-specific constraints, the algorithm loses the freedom to overfit.

  • Symmetry Constraints: Yang et al. introduce a framework in Symmetry-Informed Governing Equation Discovery that constrains the search space to equations obeying fundamental physical symmetries (such as rotational or scaling invariance). Because random noise does not respect these symmetries, the algorithm naturally ignores it.
  • Hypothesis Prompts: Bendinelli et al. treat prior scientific knowledge as a structural prompt in Controllable Neural Symbolic Regression. By "nudging" the model toward expected complexities or symmetries, they prevent it from prioritizing raw numerical accuracy over physical reality.
  • Semantic Backpropagation: Reissmann et al. ground evolutionary searches in physical logic via Constraining Genetic Symbolic Regression via Semantic Backpropagation, ensuring that the system cannot generate physically nonsensical expressions (such as adding incompatible units) simply to minimize a noise-induced residual.

2. Robust Structural Optimization

Instead of relying on genetic algorithms—which can easily propagate overfitted mutations—some modern approaches restructure the optimization process itself to be inherently noise-resilient.

For example, Kahlmeyer et al. demonstrate in Scaling Up Unbiased Search-based Symbolic Regression that a systematic, unbiased search over Directed Acyclic Graphs (DAGs) maintains simpler functional forms under noise compared to heuristic evolution.

In the domain of differential equations, calculating numerical derivatives from noisy data amplifies the noise exponentially. To bypass this, George leverages a generalized variational formulation in EDCDE - Extended Discovery of Closed-Form Differential Equations. By integrating the governing equations against smooth test functions, the variational framework mathematically integrates out the high-frequency measurement noise, allowing the system to isolate the true underlying dynamics without computing noisy derivatives.

3. Foundation Models and Reinforcement Learning Priors

A recent paradigm shift involves replacing explicit search heuristics with deep neural networks or Large Language Models (LLMs) that act as probabilistic generators.

Out of the box, zero-shot LLMs are highly sensitive to numerical noise. Park et al. show in Revisiting LLMs as Zero-Shot Time Series Forecasters: Small Noise Can Break Large Models that minor stochastic noise causes severe performance degradation in zero-shot settings. Similarly, Zheng et al. demonstrate in NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents that the symbolic reasoning capabilities of frontier LLM agents degrade significantly when faced with minimal observational variance.

However, when these models are explicitly fine-tuned with reinforcement learning (RL) on symbolic tasks, they develop a smooth, robust prior over the space of mathematical expressions.

Traditional Numerical Solvers:  Optimize purely for error reduction -> Highly sensitive to noise
Fine-tuned LLM Regressors:      Generate formulas based on learned scientific priors -> Highly stable

Lu et al. demonstrate this effect in Finetuning Large Language Model as an Effective Symbolic Regressor. By fine-tuning an LLM using RL, they created a system that maintains high predictive accuracy and structural stability in the presence of Gaussian noise, outperforming traditional numerical optimization tools that experience sharp performance declines. The neural network learns to favor clean, concise, human-like formulas, effectively acting as a regularizer that ignores random data jitters.

Key Foundational Work

BACON: A Production System That Discovers Empirical Laws (1977) — contributed 1 concept elements to this direction.
BACON.5: The Discovery of Conservation Laws (1981) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_noise_sensitivity_in_symbolic_discovery_fd9fcd7f,
  title = {noise sensitivity in symbolic discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/noise-sensitivity-in-symbolic-discovery/rdr_fd9fcd7f00c941bbb61d4bd56bca34b7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_noise_sensitivity_in_symbolic_discovery_fd9fcd7f_v1,
  title = {noise sensitivity in symbolic discovery},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/noise-sensitivity-in-symbolic-discovery/rdr_fd9fcd7f00c941bbb61d4bd56bca34b7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake