Discovery of Equations: Experimental Evaluation of Convergence

Robert Zembowicz, Jan M. ZytkowAAAI 1992

Introduction

The quest for automated scientific discovery has long been a goal of artificial intelligence. Early systems such as BACON and ABACUS demonstrated the ability to "rediscover" fundamental physical laws from data. However, these pioneering efforts often overlooked a critical aspect of scientific inquiry: experimental error. In real-world scenarios, measurements are inherently uncertain. Treating noise as a signal leads to overfitted, complex equations that lack physical meaning. This paper introduces EF (Equation Finder), a system designed to treat experimental error with the rigor of a scientist, ensuring that discovered equations converge to true physical laws as data quality improves.

Methodology: The EF System

The EF system distinguishes itself from its predecessors through a statistically grounded approach to model fitting and selection.

Statistical Rigor

EF employs weighted least-squares fitting using $\chi^2$ (chi-square) minimization. This ensures that data points with higher precision exert more influence on the final model. Crucially, the system implements error propagation; when variables are transformed (e.g., $y \rightarrow \log y$), EF uses analytical derivatives to calculate the resulting uncertainty in the new variable.

Parameter Zeroing and Model Selection

To combat overfitting, EF utilizes "parameter zeroing"—a form of Occam’s Razor. If a fitted parameter's value is smaller than its calculated error, it is set to zero, effectively pruning spurious terms. The system evaluates models based on the probability $Q$ that the observed $\chi^2$ value occurred by chance, allowing for an objective ranking of candidate equations.

Search Strategy

EF navigates a structured search space by generating new variables through a library of transformations and then fitting polynomial models to these variables.

Figure 2: The search space: generation of variables, generation of equations, and model fitting.

Experimental Evaluation

The authors evaluated EF using synthetic datasets derived from known complex source equations. This "ground truth" approach allowed for controlled testing of two dimensions of convergence:

  1. Precision Convergence: Keeping the data range constant while decreasing measurement error.
  2. Range Convergence: Keeping the error level constant while expanding the range of the independent variable.

Results and Discussion

The primary metric for success was the number of "acceptable" models found by the system. A robust discovery system should ideally converge to exactly one model—the true source equation—as data quality increases.

Figure 6: Convergence of the number of acceptable models as error level decreases across various source models.

As shown in the experimental results, EF successfully narrowed down hundreds of potential models to the single correct one as the error level dropped from 10% to 0.01%. This behavior contrasts sharply with previous systems that utilized constant system errors, which often failed to eliminate "spurious" parameters regardless of data precision.

Conclusion

By integrating systematic error propagation and statistically sound model selection, EF demonstrates that automated discovery systems can achieve scientific convergence. While challenges remain regarding combinatorial explosion in deep searches, EF provides a reliable framework for extracting meaningful empirical laws from noisy experimental data.

Summary updated

BibTeX
@inproceedings{zembowicz1992discovery,
  author = {Robert Zembowicz and Jan M. Zytkow},
  title = {{Discovery of Equations: Experimental Evaluation of Convergence}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1992},
}
Figure