ROBOTO2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment

Anthony Hevia, Sanjana Chintalapati, Veronica Ka Wai Lai, Nguyễn Thành Tâm, Wai‐Tat Wong, Terry P. Klassen, Lucy Lu Wang

TL;DR

ROBOTO2 is an open-source, web-based platform that uses Large Language Models (LLMs) to help researchers assess the "Risk of Bias" in clinical trials—a process that usually takes experts over 30 minutes per paper. Along with the tool, the authors released a benchmark dataset of 521 pediatric clinical trials, finding that while LLMs are helpful assistants, they tend to be overly "cautious" and struggle with complex medical reasoning compared to human experts.

What Problem Does This Paper Address?

In the world of evidence-based medicine, "Systematic Reviews" are the gold standard. To conduct one, researchers must evaluate the quality of every clinical trial involved using a framework called Risk of Bias (ROB). The current industry standard, Cochrane’s ROB2 tool, requires answering 22 complex "signaling questions" across five domains (e.g., was the randomization truly random? Were the results cherry-picked?).

The problem? It is incredibly labor-intensive. A single trial report can take a trained reviewer upwards of 30 minutes to assess. As the volume of medical literature explodes, this creates a massive bottleneck. Previous attempts at automation used older versions of the ROB tool or supervised models that lacked the nuanced reasoning required for the current, more rigorous ROB2 framework.

Methods

The authors developed ROBOTO2, a human-in-the-loop system designed to streamline the assessment process. The pipeline works as follows:

  1. Preprocessing: Clinical trial PDFs are parsed into structured JSON using s2orc-doc2json.
  2. Retrieval (RAG): The system uses S-BERT (Sentence-BERT) to embed paragraphs and retrieve the most relevant evidence from the paper for each specific signaling question.
  3. LLM Prompting: The retrieved evidence, the signaling question, and "elaboration text" (instructions on how to answer) are fed into an LLM. The authors tested GPT-3.5, GPT-4o, Claude 3.5-Sonnet, and Llama-3.3-70B.
  4. Interactive Review: A web interface presents the LLM’s suggested answer and rationale. Human experts can then upvote/downvote evidence, edit the explanation, or override the final judgment.

Datasets

The paper introduces a significant new benchmark:

  • 521 Pediatric Clinical Trial Reports: This dataset includes 8,954 signaling questions and 1,202 manually identified evidence passages.
  • The Split: 245 trials were assessed fully manually (serving as the "gold standard" ground truth), while 276 were assessed using the LLM-assisted ROBOTO2 workflow.
  • Availability: The dataset and code are released publicly to foster further research into automating systematic reviews.

Key Insights and Ideas

  • The "Cautious" AI: One of the most interesting findings is that LLMs are systematically more conservative than humans. They frequently opt for "High Risk" or "No Information" judgments even when evidence for "Low Risk" is present. The authors hypothesize this stems from safety and alignment training—models are essentially "afraid" to make a definitive positive claim without absolute certainty.
  • Context is King: For the most capable models (like Claude 3.5-Sonnet), providing the entire paper as context yielded better results than using Retrieval-Augmented Generation (RAG) to pick specific paragraphs. This suggests that ROB assessment requires a holistic understanding of the trial that simple keyword or semantic search might miss.
  • Domain Difficulty: LLMs perform well on Domain 1 (Randomization) because the language is standardized. However, they struggle with Domain 5 (Selection of reported results), which often requires comparing the paper against external protocols or clinical trial registries—information not always contained within the PDF itself.

Key Results

  • Top Performer: Claude 3.5-Sonnet using the full paper context achieved the highest performance with a micro-F1 score of 0.71.
  • Retrieval Efficiency: S-BERT significantly outperformed traditional BM25 keyword search, achieving a Recall@3 of 0.455 compared to 0.272.
  • Human-AI Synergy: In the assisted workflow, experts accepted the model's suggested answer 57.6% of the time and the generated rationale 71.3% of the time. This indicates that even when the model is wrong, it provides a useful "first draft" that speeds up the human reviewer.

Limitations and Future Work

  • Anchoring Bias: The authors acknowledge that showing an LLM's suggestion first might bias human experts to agree with the AI rather than thinking critically (the "LLM effect").
  • Dataset Imbalance: Like real-world clinical trials, the dataset is unbalanced; most papers fall into the "some concerns" category, with very few "low risk" examples, which can skew model training and evaluation.
  • Reasoning Models: The study was conducted before the widespread availability of "reasoning" models (like OpenAI’s o1 or DeepSeek-R1). The authors suggest that these models, which use chain-of-thought processing, might be better suited for the complex logic required by the ROB2 tool.

Summary updated

BibTeX
@misc{hevia2025roboto,
  author = {Anthony Hevia and Sanjana Chintalapati and Veronica Ka Wai Lai and Nguyễn Thành Tâm and Wai‐Tat Wong and Terry P. Klassen and Lucy Lu Wang},
  title = {{ROBOTO2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment}},
  year = {2025},
  eprint = {2511.03048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.03048},
}
Figure