Advancing AI Research Assistants with Expert-Involved Learning

Tianyu Liu, Simeng Han, Xiao Luo, Hanchen Wang, Pan Lu, Biqing Zhu, Yuge Wang, Keyi Li, Jiapeng Chen, Rihao Qu, Yufeng Liu, Cui Xinyue, Aviv Yaish, Yuhang Chen, Minsheng Hao, Chuhan Li, Kangkang Li, Arman Cohan, Hua Xu, Mark Gerstein, James Zou, Hongyu Zhao

Abstract: Large language models (LLMs) and large multimodal models (LMMs) promise to accelerate biomedical discovery, yet their reliability remains unclear. We introduce ARIEL (AI Research Assistant for Expert-in-the-Loop Learning), an open-source evaluation and optimization framework that pairs a curated multimodal biomedical corpus with expert-vetted tasks to probe two capabilities: full-length article summarization and fine-grained figure interpretation. Using uniform protocols and blinded PhD-level evaluation, we find that state-of-the-art models generate fluent but incomplete summaries, whereas LMMs struggle with detailed visual reasoning. We later observe that prompt engineering and lightweight fine-tuning substantially improve textual coverage, and a compute-scaled inference strategy enhances visual question answering. We build an ARIEL agent that integrates textual and visual cues, and we show it can propose testable mechanistic hypotheses. ARIEL delineates current strengths and limitations of foundation models, and provides a reproducible platform for advancing trustworthy AI in biomedicine.

Introduction

The explosion of biomedical literature has created a "data deluge" that human researchers struggle to navigate. While Large Language Models (LLMs) and Multimodal Models (LMMs) promise to help by summarizing papers and interpreting figures, their reliability in specialized domains remains a major concern. Existing benchmarks often rely on simple multiple-choice questions that do not reflect the actual complexity of scientific work. ARIEL (AI Research Assistant for Expert-in-the-Loop Learning) addresses this "reliability gap" by creating a rigorous, expert-vetted environment to test if AI can truly function as a reliable research assistant.

The ARIEL Framework

ARIEL focuses on two core pillars: full-length article summarization and fine-grained figure interpretation. The framework moves beyond simple prompting by employing specialized optimization strategies.

Overview of the ARIEL framework, illustrating the pipeline from candidate models and datasets to expert-led evaluation and improvement strategies.

Datasets and Methodology

To ensure rigorous testing and avoid "data leakage," the study introduced two primary datasets consisting of content published post-January 2024:

  • Manuscript Dataset: 2,571 recent PubMed papers covering topics like COVID-19, SARS-CoV-2, and general healthcare.
  • Figure Dataset: 10 high-quality, complex figures from top-tier journals, paired with 88 open-ended questions requiring deep reasoning.
  • Human-Eval: A "gold standard" set of evaluations provided by 12 doctoral-level biomedical researchers.

Improvement strategies included Low-Rank Adaptation (LoRA) for lightweight fine-tuning and test-time computation scaling (notably with OpenAI’s o1 model), which allows models to explore multiple reasoning paths before answering.

Key Results and Insights

Evaluation across various metrics revealed that while foundation models are highly fluent, they often miss critical numerical data or specific experimental regimens. However, targeted optimization significantly boosts performance.

Comprehensive evaluation results comparing various LLMs and LMMs across textual and visual metrics, highlighting the superior performance of fine-tuned models.
  • Fine-tuning vs. Prompting: For long-form scientific summarization, lightweight fine-tuning (LoRA) was far more effective than prompt engineering. The fine-tuned ChatGLM4_lora model significantly outperformed human participants in summarization metrics (p-value = 0.015).
  • The Power of "Thinking Time": Standard LMMs struggle with complex figures, but models using test-time reasoning (like o1) showed a massive jump in accuracy, surpassing other LMMs and even some human experts.
  • AI as a Collaborative Editor: In verification-correction experiments, ARIEL improved the accuracy of human participants by over 65% in cases where the human's initial analysis was weak.
  • Economic Efficiency: Summarizing a paper cost a human expert roughly $365 in labor time, compared to just $0.02 for the model, representing a staggering increase in efficiency.

Conclusion and Future Work

ARIEL demonstrates that while standard models have limitations in biomedical reasoning, specialized strategies like fine-tuning and extended inference can push AI to perform at expert levels. Future work will expand the framework into more niche biomedical sub-fields and explore "agentic" workflows where multiple specialized AI agents collaborate to solve complex research problems. The current "HypoGenAgent" already shows promise in proposing testable mechanistic hypotheses from visual data that are rated highly for novelty and significance.

Summary updated

Abstract

Large language models (LLMs) and large multimodal models (LMMs) promise to accelerate biomedical discovery, yet their reliability remains unclear. We introduce ARIEL (AI Research Assistant for Expert-in-the-Loop Learning), an open-source evaluation and optimization framework that pairs a curated multimodal biomedical corpus with expert-vetted tasks to probe two capabilities: full-length article summarization and fine-grained figure interpretation. Using uniform protocols and blinded PhD-level evaluation, we find that state-of-the-art models generate fluent but incomplete summaries, whereas LMMs struggle with detailed visual reasoning. We later observe that prompt engineering and lightweight fine-tuning substantially improve textual coverage, and a compute-scaled inference strategy enhances visual question answering. We build an ARIEL agent that integrates textual and visual cues, and we show it can propose testable mechanistic hypotheses. ARIEL delineates current strengths and limitations of foundation models, and provides a reproducible platform for advancing trustworthy AI in biomedicine.
BibTeX
@misc{liu2025advancing,
  author = {Tianyu Liu and Simeng Han and Xiao Luo and Hanchen Wang and Pan Lu and Biqing Zhu and Yuge Wang and Keyi Li and Jiapeng Chen and Rihao Qu and Yufeng Liu and Cui Xinyue and Aviv Yaish and Yuhang Chen and Minsheng Hao and Chuhan Li and Kangkang Li and Arman Cohan and Hua Xu and Mark Gerstein and James Zou and Hongyu Zhao},
  title = {{Advancing AI Research Assistants with Expert-Involved Learning}},
  year = {2025},
  eprint = {2505.04638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.04638},
}
Figure