Abstract: Large language models (LLMs) and large multimodal models (LMMs) promise to accelerate biomedical discovery, yet their reliability remains unclear. We introduce ARIEL (AI Research Assistant for Expert-in-the-Loop Learning), an open-source evaluation and optimization framework that pairs a curated multimodal biomedical corpus with expert-vetted tasks to probe two capabilities: full-length article summarization and fine-grained figure interpretation. Using uniform protocols and blinded PhD-level evaluation, we find that state-of-the-art models generate fluent but incomplete summaries, whereas LMMs struggle with detailed visual reasoning. We later observe that prompt engineering and lightweight fine-tuning substantially improve textual coverage, and a compute-scaled inference strategy enhances visual question answering. We build an ARIEL agent that integrates textual and visual cues, and we show it can propose testable mechanistic hypotheses. ARIEL delineates current strengths and limitations of foundation models, and provides a reproducible platform for advancing trustworthy AI in biomedicine.
Introduction
The explosion of biomedical literature has created a "data deluge" that human researchers struggle to navigate. While Large Language Models (LLMs) and Multimodal Models (LMMs) promise to help by summarizing papers and interpreting figures, their reliability in specialized domains remains a major concern. Existing benchmarks often rely on simple multiple-choice questions that do not reflect the actual complexity of scientific work. ARIEL (AI Research Assistant for Expert-in-the-Loop Learning) addresses this "reliability gap" by creating a rigorous, expert-vetted environment to test if AI can truly function as a reliable research assistant.
The ARIEL Framework
ARIEL focuses on two core pillars: full-length article summarization and fine-grained figure interpretation. The framework moves beyond simple prompting by employing specialized optimization strategies.
Datasets and Methodology
To ensure rigorous testing and avoid "data leakage," the study introduced two primary datasets consisting of content published post-January 2024:
- Manuscript Dataset: 2,571 recent PubMed papers covering topics like COVID-19, SARS-CoV-2, and general healthcare.
- Figure Dataset: 10 high-quality, complex figures from top-tier journals, paired with 88 open-ended questions requiring deep reasoning.
- Human-Eval: A "gold standard" set of evaluations provided by 12 doctoral-level biomedical researchers.
Improvement strategies included Low-Rank Adaptation (LoRA) for lightweight fine-tuning and test-time computation scaling (notably with OpenAI’s o1 model), which allows models to explore multiple reasoning paths before answering.
Key Results and Insights
Evaluation across various metrics revealed that while foundation models are highly fluent, they often miss critical numerical data or specific experimental regimens. However, targeted optimization significantly boosts performance.
- Fine-tuning vs. Prompting: For long-form scientific summarization, lightweight fine-tuning (LoRA) was far more effective than prompt engineering. The fine-tuned ChatGLM4_lora model significantly outperformed human participants in summarization metrics (p-value = 0.015).
- The Power of "Thinking Time": Standard LMMs struggle with complex figures, but models using test-time reasoning (like o1) showed a massive jump in accuracy, surpassing other LMMs and even some human experts.
- AI as a Collaborative Editor: In verification-correction experiments, ARIEL improved the accuracy of human participants by over 65% in cases where the human's initial analysis was weak.
- Economic Efficiency: Summarizing a paper cost a human expert roughly $365 in labor time, compared to just $0.02 for the model, representing a staggering increase in efficiency.
Conclusion and Future Work
ARIEL demonstrates that while standard models have limitations in biomedical reasoning, specialized strategies like fine-tuning and extended inference can push AI to perform at expert levels. Future work will expand the framework into more niche biomedical sub-fields and explore "agentic" workflows where multiple specialized AI agents collaborate to solve complex research problems. The current "HypoGenAgent" already shows promise in proposing testable mechanistic hypotheses from visual data that are rated highly for novelty and significance.
Summary updated