ROBoto2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment

Anthony Hevia, Sanjana Shivani Chintalapati, Veronica Ka Wai Lai, Thanh Tam Nguyen, Wai-Tat Wong, Terry Klassen, Lucy Lu Wang

Introduction

In evidence-based medicine, systematic reviews are the primary method for synthesizing clinical evidence. A critical, yet labor-intensive, component of these reviews is the Risk of Bias (ROB) assessment—determining if a study's results are trustworthy or compromised by design flaws. The current standard, the Cochrane ROB2 tool, involves 22 signaling questions across five domains. For human experts, assessing a single trial can take upwards of 30 minutes, creating a significant bottleneck in medical research.

Methodology

We introduce ROBOTO2, an open-source, web-based platform designed to accelerate ROB assessments through a Retrieval-Augmented Generation (RAG) pipeline and a human-in-the-loop interface.

The ROBOTO2 system architecture, illustrating the preprocessing and retrieval/QA modules.

The system processes clinical trial reports through a four-stage pipeline:

  1. PDF Parsing: Converting trial reports into structured JSON using s2orc-doc2json.
  2. Retrieval: Utilizing Sentence-Transformers (all-MiniLM-L6-v2) to identify relevant paragraphs for each signaling question.
  3. LLM Prompting: Feeding retrieved context into Large Language Models (LLMs) like Claude 3.5-Sonnet to generate preliminary answers and rationales.
  4. Interactive Review: A web interface allowing experts to review, edit, and finalize judgments.

The ROBOTO2 Dataset

The project introduces a significant new benchmark: the ROBOTO2 Dataset. This collection includes 521 pediatric clinical trial reports with 8,954 signaling questions and 1,202 manually identified evidence passages. The dataset facilitates direct comparison between fully manual assessments (245 trials) and LLM-assisted workflows (276 trials).

Results and Human-AI Synergy

Benchmarking revealed that Claude 3.5-Sonnet achieved the highest performance, with a Micro-F1 of 0.71 and a Macro-F1 of 0.82 when utilizing the full paper text as context. Semantic retrieval via S-BERT significantly outperformed traditional keyword-based methods.

The ROBOTO2 interactive interface, displaying retrieved evidence, LLM-generated predictions, and expert validation controls.

The system demonstrates effective human-AI synergy. Experts accepted model-generated rationales 71.3% of the time. Even when the AI's final judgment required correction, its ability to locate and summarize relevant text provided a usable starting point for the majority of the work, with experts providing entirely original answers only 42.4% of the time.

Discussion and Limitations

Analysis indicates that LLMs are often "conservative," frequently opting for "High Risk" or "No Information" judgments to avoid hallucinations. While models excel in domains with standardized language (e.g., Randomization), they struggle with tasks requiring external world knowledge, such as comparing reports against external protocols.

Future work will address "anchoring bias"—the psychological risk of experts over-relying on AI suggestions—and improve numerical reasoning for metrics like attrition rates. The integration of newer reasoning-focused models is expected to further enhance performance on complex logical assessments.

Summary updated

BibTeX
@misc{hevia2025roboto,
  author = {Anthony Hevia and Sanjana Shivani Chintalapati and Veronica Ka Wai Lai and Thanh Tam Nguyen and Wai-Tat Wong and Terry Klassen and Lucy Lu Wang},
  title = {{ROBoto2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment}},
  year = {2025},
  eprint = {2511.03048},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.03048},
}
Figure