Abstract
Clinical trial recruitment is a significant bottleneck in medical research, with approximately 80% of trials failing to meet enrollment targets on time. This paper introduces TRIALQA, the first expert-annotated dataset designed to evaluate Large Language Models (LLMs) in identifying potential research participants from social media signals. Our evaluation of seven open-source models reveals that while fine-tuning can achieve up to 85% accuracy in identifying research interest, LLMs still struggle with the complex, multi-hop reasoning required to verify specific medical eligibility criteria in sparse, self-reported health data.
Introduction
The recruitment crisis in clinical trials is a major hurdle for medical advancement, often leading to trial termination due to poor accrual. Traditional recruitment methods, such as hospital record reviews and direct mailings, are frequently slow and geographically limited.
Concurrently, platforms like Reddit have become a significant source of self-reported health data where users openly discuss diagnoses, treatments, and medical concerns. This research explores whether LLMs can be leveraged to "listen" to these social media signals to automatically identify individuals who may be both eligible for and interested in specific clinical trials.
Methodology
To evaluate LLM capabilities, the researchers benchmarked seven open-source models—including Llama 3, Qwen 2.5, DeepSeek, and Mistral-7B—against a RoBERTa baseline. The study explored several prompting and training strategies:
- Direct Inference: Simple True/False/Unknown classification.
- In-Context Learning (ICL): Few-shot prompting with examples.
- Chain-of-Thought (CoT): Step-by-step reasoning prompts.
- Self-Consistency: Majority voting across multiple generation paths.
- Fine-tuning (LoRA): Task-specific training at both the "entry-level" (individual posts) and "user-level" (entire post history).
The TRIALQA Dataset
The core contribution of this work is TRIALQA, a novel dataset curated from the r/coloncancer and r/prostatecancer subreddits. Expert annotators with medical and computer science backgrounds labeled users against eight eligibility criteria (e.g., age, history of radiation) and six "interest reasons" (e.g., seeking new therapies).
The dataset captures the nuances of social media communication, where medical details are often omitted, leading to a high frequency of "Unknown" labels. This sparsity presents a significant challenge for automated classification.
Key Insights and Ideas
The "Unknown" Default
A primary hurdle is the "thinness" of social media data. Models frequently default to an "Unknown" label because users rarely explicitly state negative criteria (e.g., "I have not taken antibiotics in the last two weeks").
Model Size vs. Performance
Surprisingly, smaller models like Mistral-7B often outperformed 70B-parameter giants in direct inference. The authors attribute this to larger models being "too verbose" or failing to strictly follow the required output format, which interfered with automated evaluation.
Reasoning vs. Conclusion
A specific failure mode was identified where models performed correct logical reasoning (e.g., correctly identifying that a post discussed a family member's cancer rather than the user's) but ultimately provided an incorrect final eligibility conclusion.
Key Results
Fine-tuning via Low-Rank Adaptation (LoRA) provided the most significant performance gains. For colon cancer, entry-level fine-tuning achieved an accuracy of 0.70 for eligibility criteria and a high 0.85 for identifying research interest.
Identifying "Interest Reasons" (binary: True/False) proved to be a significantly easier task for the models than verifying "Eligibility Criteria" (three-class: True/False/Unknown), which requires deeper medical context and multi-hop reasoning.
Limitations and Future Work
The study has several constraints, including platform bias (Reddit-only data) and a limited geographic scope (two U.S.-based trials). Furthermore, the sensitive nature of health data highlights the ongoing tension between open science and patient privacy.
Future work will focus on expanding to other platforms like Facebook and Patient.info, and developing better techniques to help LLMs connect disparate clues across a user's entire posting history to improve multi-hop reasoning capabilities.
Summary updated