AI-Assisted Scientific Data Collection with Iterative Human Feedback

Travis Mandel, James Boyd, Sebastian J. Carter, Randall H. Tanaka, Taishi NammotoAAAI 2021 · DOI

Introduction

Traditional AI-driven data collection frameworks, such as active learning or reinforcement learning, typically operate under pre-defined objective functions. While efficient for known goals, this approach often fails to capture the serendipitous nature of scientific discovery. Historical breakthroughs, like the discovery of penicillin, frequently stem from researchers noticing unexpected anomalies that fall outside original experimental designs.

To bridge this gap, we present a framework for human-AI collaboration that allows scientists to identify interesting patterns as they emerge. The core of this framework is the Threshold Estimating Sampling Algorithm (TESA), a zero-regret algorithm designed to prioritize sampling in scientifically promising areas based on iterative human feedback.

Methodology

TESA facilitates a dynamic interaction where a human researcher periodically provides feedback by marking "keypoints"—specific data regions that appear scientifically significant. This addresses the fundamental explore-exploit dilemma: determining whether the AI should gather more data at a known keypoint to increase statistical confidence or explore seemingly uninteresting areas for new discoveries.

Technical Innovations

  • Bayesian Thresholding: TESA models the human decision-making process by assuming a threshold ($h$) of samples ($n_i$) is required for a point to be deemed interesting. Using a Pareto distribution to estimate this threshold, the algorithm can effectively prune the search space by stopping sampling in areas that exceed the threshold without being marked as keypoints.
  • Zero-Regret Guarantee: The framework introduces a novel definition of "external regret" for human-in-the-loop settings. TESA is mathematically proven to achieve zero regret asymptotically, ensuring long-term optimal performance.
  • Iterative Feedback Loop: Unlike static models like Uncertainty Weighted Posterior Sampling (UWPS), TESA allows humans to add or modify keypoints every $I$ timesteps, accommodating the evolving intuition of a researcher as data is collected.
Cumulative regret comparison across different sampling strategies. TESA variants (0, 10, 20) demonstrate significantly lower regret than traditional baselines.

Experimental Evaluation

TESA was evaluated using "replay" simulations across three real-world scientific datasets: Economics (avocado price vs. sales), Mental Health (tech worker demographics vs. diagnosis rates), and Cognitive Psychology (stimulus size vs. reaction time). Additionally, a human subject experiment involving 101 participants was conducted to compare TESA against traditional workflows requiring upfront "prior theories."

Results and Discussion

In all simulated scientific domains, TESA variants consistently outperformed standard baselines, including Round Robin, Epsilon-Greedy, and Thompson Sampling. Notably, TESA significantly surpassed UWPS, which relies on static prior predictions rather than dynamic feedback.

Performance comparison in a secondary experimental setting, highlighting TESA's consistent superiority over active learning and reinforcement learning methods.

Key Insights

  • Dynamic Interest: Scientific interest is non-stationary. TESA effectively manages this by allowing for the re-evaluation of data points as more information becomes available.
  • Modeling the Human: By estimating the "human threshold" for interest, TESA prunes the search space more effectively than algorithms that only model the data itself.
  • User Preference: In the human subject study, 95% of participants who passed comprehension checks preferred TESA's iterative data collection over the prior-theory method, citing that TESA-collected data was more "scientifically interesting."

Conclusion and Future Work

While TESA shows great promise for 1D response functions, future work will address higher-dimensional spaces and physical constraints, such as the movement costs of robotic sampling systems. Additionally, while asymptotic zero-regret is proven, establishing specific convergence rates remains a priority for theoretical advancement. TESA represents a significant step toward AI systems that truly augment human scientific intuition rather than just automating pre-defined tasks.

Summary updated

BibTeX
@inproceedings{mandel2021ai,
  author = {Travis Mandel and James Boyd and Sebastian J. Carter and Randall H. Tanaka and Taishi Nammoto},
  title = {{AI-Assisted Scientific Data Collection with Iterative Human Feedback}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i7.16744},
}
Figure