Introduction
The quest for automated scientific discovery has long been a cornerstone of AI research. Early systems like Bacon and Abacus aimed to uncover mathematical laws from numerical data by searching through an expansive space of potential formulas. However, these pioneering efforts often struggled with "overfitting"—identifying complex but spurious patterns in noisy data. Cullen Schaffer's work challenges this "infinite search" paradigm, proposing that the primary hurdle in scientific discovery is not the generation of hypotheses, but their rigorous evaluation. By reframing function-finding as a classification problem among a small set of likely candidates, Schaffer introduces the E* algorithm, prioritizing reliability over raw search power.
The E* Methodology
The E algorithm departs from traditional heuristic search methods in three fundamental ways. First, it operates on a fixed candidate set. Rather than exploring an open-ended mathematical space, E considers only eight possibilities: a linear relationship ($y = k_1x + k_2$), six specific power proportionalities ($y = kx^n$ for $n \in \{-2, -1, -0.5, 0.5, 1, 2\}$), or a "No relationship" null hypothesis. This constraint is based on the observation that a vast majority of published scientific relationships follow these simple forms.
Second, E employs three rigorous evaluation criteria* to vet these candidates:
- Significance: Utilizing statistical measures like t-statistics to ensure a pattern is unlikely to be a product of random chance.
- Distinction: Assessing the margin by which the best-fitting model outperforms its competitors. If multiple models fit similarly well, the algorithm remains conservative.
- Systematic Lack of Fit: E* scrutinizes the residuals (errors) of a fit. If the residuals exhibit a non-random, systematic pattern—such as a parabolic curve—the model is rejected as fundamentally flawed, even if its overall correlation appears high.
Finally, E was developed using prospective testing*. The algorithm's parameters were "locked" after a preliminary development phase, ensuring that its performance on subsequent test data was a true measure of its generalizability rather than the result of iterative tuning by the researcher.
Empirical Evaluation
To validate E, Schaffer compiled a substantial benchmark of 352 real-world scientific datasets. A key component of this was the Physical Review* collection, consisting of bivariate data systematically extracted from early 20th-century issues of the journal. This provided a diverse and authentic testing ground, moving beyond the hand-selected, often artificial datasets used in previous studies. The evaluation used 100 cases (comprising 192 datasets) for a rigorous prospective test, where the "correct" answer was defined as the relationship originally hypothesized by the publishing scientists.
Comparative Performance
The effectiveness of E was measured against various configurations of the Bacon* algorithm, the industry standard at the time. The results highlight a significant breakthrough in reliability.
As shown in the table, E achieved 31.50 correct identifications with only 10.16 incorrect answers. In contrast, for the Bacon algorithm to achieve a comparable number of correct answers (33.00), it produced 31.33 incorrect answers—a three-fold increase in "false leads." This demonstrates that E effectively manages the reliability-power tradeoff, providing a much safer tool for scientists by significantly reducing the cost of pursuing spurious patterns.
Conclusion
The E algorithm demonstrates that a simplified, classification-based approach to function-finding can vastly outperform complex search heuristics in terms of reliability. By focusing on a small set of scientifically plausible models and applying rigorous statistical checks, E provides a robust framework for automated discovery. While currently limited to bivariate relationships and fixed functional forms, the success of E* suggests that the future of AI in science lies in sophisticated evaluation rather than unconstrained search. It proves that in the pursuit of scientific truth, the ability to say "I don't know" is often more valuable than the ability to find a pattern in everything.
Summary updated