LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs

Sanchit Kabra, Nikhil Abhyankar, Saaketh Desai, Prasad Iyer, Chandan K ReddyarXiv 2026 · arXiv

TL;DR

LLM-AutoSciLab is a closed-loop framework that transforms AI-driven scientific discovery from passive data fitting into active, hypothesis-guided experimentation. By designing experiments specifically to resolve disagreements between competing hypotheses, it recovers true underlying scientific laws and causal graphs 2–5× more sample-efficiently than traditional active learning baselines.

What Problem Does This Paper Address?

Most machine learning approaches to scientific discovery treat it as a static, supervised learning problem over fixed datasets. However, real-world data is often limited and can be explained equally well by multiple different mathematical models—a challenge known as the identifiability bottleneck. While these models might fit existing data locally, they often fail to generalize because they don't capture the true underlying mechanism.

True scientific discovery is an iterative, closed-loop process. Scientists don't just passively fit data; they formulate hypotheses and actively design targeted experiments to force competing explanations to diverge. This paper addresses the lack of automated systems capable of mimicking this hypothesis-driven data acquisition process across diverse scientific domains.

Methods

The authors introduce LLM-AutoSciLab, an automated framework that couples large language models (LLMs) with domain-specific optimization tools to run a complete experimental loop:

  1. Hypothesis Generation: To prevent the system from fixating on a single idea too early, the framework decouples exploration from synthesis. A smaller LLM generates a diverse batch of candidate hypotheses, which are clustered into structural families. A larger LLM then reviews these families to propose a primary hypothesis, alternative candidates, and recommended experimental search regions.
  2. Hypothesis-Conditioned Experiment Selection: Instead of selecting experiments to minimize overall predictive uncertainty (the standard approach in active learning), LLM-AutoSciLab targets regions where the candidate hypotheses disagree the most. This prioritizes mechanism discrimination.
  3. Hypothesis Optimization & Confidence Feedback: New experimental observations are fed into domain-specific backends (like PySR for symbolic regression or BFGS for graph discovery) to refine the models. A "Bootstrap Confidence Gate" evaluates the stability of the current best model. If confidence is low, the system stays in a "Disambiguate" mode to separate competing theories; once confidence crosses a threshold, it switches to a "Refine" mode to fine-tune parameters.
Figure 1: Overview of LLM-AutoSciLab. (A) An LLM generates candidate hypotheses from observations and memory. (B) Experiments are actively selected in regions of maximal disagreement with the hypothesis. (C) Candidates are iteratively refined via domain-specific optimization (e.g., parameter fitting and constraint enforcement), with confidence-based feedback guiding updates.

Datasets

To evaluate active, budget-constrained discovery, the authors introduce ActiveSciBench, a new benchmark suite that hides relevant variables and requires active exploration:

  • ActiveSciBench-Chem: Consists of 57 enzyme-kinetics tasks where the model must discover symbolic rate laws from a 7-dimensional interface containing distractor variables.
  • ActiveSciBench-GRN: Consists of 45 gene-regulatory-network tasks requiring the recovery of signed, directed causal graphs from discrete perturbation-response experiments.

They also evaluate their method on NewtonBench, an existing benchmark featuring 12 physics domains used to test symbolic law recovery when relevant variables are already known.

Table 2: Comparison of scientific discovery benchmarks across key properties.

Key Insights and Ideas

  • Falsification Beats Uncertainty Reduction: Traditional active learning reduces statistical uncertainty, but LLM-AutoSciLab demonstrates that actively trying to falsify and differentiate explicit candidate mechanisms is far more effective for discovering true physical laws under tight budgets.
  • LLMs Need Downstream Anchors: Pure LLM prompting or code-generation baselines struggle with scientific discovery because they default to generic textbook templates rather than testing variable relevance. True discovery requires coupling LLM structural reasoning with rigorous downstream numerical optimization.
  • Overfitting the Data vs. Finding the Law: Standard symbolic regression tools can achieve high numerical accuracy by adding spurious terms (like unnecessary sinusoidal components) that fit the local data but destroy the underlying physical meaning. Guided experimentation prevents this structural drift.

Key Results

LLM-AutoSciLab significantly outperformed all baseline methods across all three benchmarks:

  • NewtonBench: Achieved 67.6% symbolic accuracy, vastly outperforming standard PySR (24.07%), which frequently overfit the data by adding spurious terms.
  • ActiveSciBench-Chem: Achieved 35.1% symbolic accuracy, whereas traditional Bayesian Experimental Design collapsed on harder tasks (0.0% accuracy).
  • ActiveSciBench-GRN: Achieved 31.1% exact graph recovery and a 72.5% Edge F1 score, outperforming specialized causal discovery baselines like GIES (6.67% exact graph recovery).
  • Sample Efficiency: The framework proved to be 2–5× more sample-efficient than competing baselines, reaching the target accuracy with a fraction of the experimental budget.
Figure 3: Qualitative NewtonBench case study. LLM-AutoSciLab recovers the correct symbolic structure, while other baselines introduce spurious terms, collapse to incorrect families, or recover only simplified harmonic forms.

Limitations and Future Work

  • Simulator-Based Oracles: The current framework relies entirely on clean, simulated environments. It does not yet account for real-world physical lab challenges such as measurement noise, experimental failures, execution costs, or complex operational constraints.
  • Dependency on Backends: The success of the framework is tightly bound to the quality of the LLM's initial hypotheses and the capabilities of the underlying refinement tools (e.g., PySR). Expanding to broader scientific domains will require richer optimization backends and validation in real-world wet labs.

Summary updated

Abstract

Scientific discovery is a closed-loop process in which hypotheses guide data acquisition and observations refine the hypothesis space. Yet most approaches reduce discovery to supervised learning over fixed datasets, where limited observations can support multiple plausible mechanisms that fit locally but fail to generalize. Thus, the key challenge is selecting informative observations to resolve uncertainty, shifting the focus from static inference to adaptive data acquisition. To address this, we propose LLM-AutoSciLab, a closed-loop framework that couples hypothesis generation with hypothesis-conditioned experiment selection and mechanism refinement. Rather than fitting models to passively collected data, LLM-AutoSciLab iteratively proposes plausible hypotheses, selects informative experiments to distinguish or refine them, and updates its state using the resulting evidence. To evaluate dynamic, closed-loop scientific discovery with active data acquisition, we introduce ActiveSciBench, comprising two datasets: ActiveSciBench-Chem with 57 enzyme-kinetics tasks and ActiveSciBench-GRN with 45 gene-regulatory-network tasks. These datasets model discovery as a budget-constrained process requiring adaptive experiment design, variable selection, and recovery of true mechanisms. Across NewtonBench, ActiveSciBench-Chem, and ActiveSciBench-GRN, LLM-AutoSciLab outperforms prior methods, achieving 67.6% and 35.1% symbolic accuracy on NewtonBench and ActiveSciBench-Chem, respectively, and 31.1% exact graph recovery on ActiveSciBench-GRN. Moreover, hypothesis-guided experimentation is 2-5x more sample-efficient than the strongest competing baselines. Code and data are available at: https://github.com/scientific-discovery/LLM-AutoSciLab
BibTeX
@misc{kabra2026llm,
  author = {Sanchit Kabra and Nikhil Abhyankar and Saaketh Desai and Prasad Iyer and Chandan K Reddy},
  title = {{LLM-AutoSciLab: Closed-Loop Scientific Discovery via Active Experimentation with LLMs}},
  year = {2026},
  eprint = {2605.24043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.24043},
}

Loading related content...

Figure