Abstract
If machine learning and other automation technologies are to be used more widely and to their full potential in systematic reviews and guideline development, it is crucial to ensure new technologies are in line with current values and practice. It will also be important to maximize the transparency of the methods of these technologies to address the concerns of guideline developers.
Introduction
The rapid expansion of medical research has created a "data deluge," with approximately 4,000 new health research articles published daily. For organizations responsible for clinical guidelines, such as the World Health Organization (WHO) or the National Institute for Health and Care Excellence (NICE), maintaining up-to-date evidence through manual systematic reviews has become an increasingly insurmountable challenge.
While automation technologies like Natural Language Processing (NLP) and Machine Learning (ML) offer potential solutions for tasks like screening, their adoption remains sluggish. This study investigates the perspectives of guideline developers—the primary "gatekeepers" of evidence—to understand the barriers to integrating these technologies into professional practice.
Methodology
This qualitative study utilized Rogers’ Diffusion of Innovations framework to analyze the adoption of automation. The framework evaluates innovations across five key dimensions:
- Relative Advantage: The perceived superiority over existing methods.
- Compatibility: Alignment with existing values, experiences, and needs.
- Complexity: The degree of perceived difficulty in understanding and use.
- Trialability: The ability to experiment with the innovation on a limited basis.
- Observability: The visibility of the results to others.
The researchers conducted 18 semi-structured interviews with professionals from prominent health organizations across Australia, the UK, and the USA. Participants represented diverse backgrounds with experience levels ranging from under five to over twenty years. Data analysis followed a five-stage thematic approach, combining deductive mapping to the framework with inductive grounded theory to capture emergent sub-themes.
Results and Discussion
The analysis revealed that technical performance is secondary to professional alignment. The following themes emerged as critical to adoption:
The Primacy of Compatibility
"Compatibility" was the most dominant theme, appearing in approximately 45% of the thematic coverage. Guideline developers do not merely seek functional tools; they require systems that respect the rigorous, human-centric nature of evidence synthesis. A core requirement is the "human-in-the-loop" model, where automation serves as a preliminary filter that must be verified by human experts to ensure the "trustworthiness" of the final output.
Transparency as a Prerequisite for Accountability
"Observability" (25% coverage) was closely linked to transparency. Because guideline developers are publicly accountable for their recommendations, they expressed a fundamental need to understand the underlying logic of algorithmic decisions. "Black box" systems are viewed with skepticism; developers require the ability to audit and defend the evidence base.
Redefining Relative Advantage
While "Relative Advantage" accounted for 25% of the discussion, it was framed less as a cost-cutting measure and more as a means of resource redirection. Automation is seen as a way to liberate human experts from rote tasks, allowing them to focus on complex qualitative judgments and nuanced evidence grading.
Complexity and Trialability
Interestingly, "Complexity" and "Trialability" were minor concerns (approximately 5% each). This suggests that the technical learning curve is not the primary bottleneck. Developers are willing to engage with sophisticated software provided it aligns with their ethical and professional standards.
Limitations and Future Directions
The study's findings are limited by a relatively small sample size (n=18) and a geographic bias toward Australian professionals. Furthermore, many participants had limited direct experience with machine learning, which may influence their perceptions toward general skepticism.
Future efforts to integrate automation into health evidence synthesis should prioritize validation and transparency over raw accuracy. Developers of these tools should focus on "explainable AI" features that facilitate human oversight. By positioning technology as a collaborative partner rather than a replacement, the field can better address the data deluge while maintaining the high standards required for clinical guidelines.
Summary updated