ADVISE: AI-accelerated Design of Evidence Synthesis for Global Development

Kristen M. Edwards, Binyang Song, Jaron Porciello, Mark Engelbert, Carolyn Huang, Faez AhmedarXiv 2023 · DOI · arXiv

Introduction

In the field of global development, policy-makers rely on Evidence Synthesis to transform vast quantities of research into actionable insights. A cornerstone of this process is the Evidence Gap Map (EGM)—a visual matrix that maps specific "interventions" (such as water management or improved seeds) against "outcomes" (like profit or income). By identifying where evidence is plentiful and where "gaps" exist, EGMs guide future research and funding.

An example of an Evidence Gap Map (EGM) visualizing the relationship between interventions and outcomes.

The Bottleneck: Information Explosion

The primary challenge facing evidence synthesis is the sheer volume of scientific literature. With over 2.6 million articles published annually, creating a single EGM can require human experts to manually screen tens of thousands of titles and abstracts. This manual process often takes six months or more, creating a significant bottleneck in the evidence-to-policy pipeline. While basic machine learning tools like Support Vector Machines (SVMs) have been used, they often lack the linguistic nuance required for complex development topics.

The ADVISE Solution

To address this, researchers developed ADVISE, a human-AI hybrid workflow designed to accelerate the most labor-intensive stages of evidence synthesis.

The standard six-step evidence synthesis workflow, where ADVISE focuses on accelerating the screening of titles and abstracts.

The ADVISE system integrates three core components:

  1. BERT-based Classification: Utilizing Bidirectional Encoder Representations from Transformers (BERT), the system understands context and semantics far better than traditional keyword-based or SVM models.
  2. Active Learning (AL): The AI interacts dynamically with human screeners. As humans label small batches of papers, the AI retrains itself to better predict relevance. Researchers found that a training set of roughly 5,000 documents offers the optimal balance between model performance and human effort.
  3. Prioritized Sampling: ADVISE employs a Highest Priority (HP) sampling strategy. Instead of focusing on papers the model is "unsure" about (Least Confidence sampling), it surfaces the documents most likely to be relevant first, allowing humans to find the "needles in the haystack" much faster.

Key Results and Impact

The implementation of ADVISE has demonstrated transformative potential for the sector:

  • Significant Labor Reduction: The BERT-based agent alone reduced human screening effort by 68.5%. When combined with the HP Active Learning strategy, the total reduction in effort reached 78.3% for identifying 80% of relevant documents.
  • Superior Performance: BERT outperformed industry-standard SVM tools by 12% in accuracy and saved 16.8% more labor.
  • Real-world Scalability: In a typical agriculture-focused EGM involving over 200,000 papers, the workflow is estimated to save a single expert approximately 1,111.5 hours of manual labor.

Future Directions

While ADVISE significantly streamlines the title and abstract screening phase, challenges remain. The "cold start" problem—where the AI requires initial human input to understand a new domain—remains a hurdle. Future work will explore the automation of full-text PDF screening and the integration of Large Language Models (LLMs) to provide qualitative summaries alongside quantitative mapping.

Summary updated

BibTeX
@misc{edwards2023advise,
  author = {Kristen M. Edwards and Binyang Song and Jaron Porciello and Mark Engelbert and Carolyn Huang and Faez Ahmed},
  title = {{ADVISE: AI-accelerated Design of Evidence Synthesis for Global Development}},
  year = {2023},
  eprint = {2305.01145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.01145},
}
Figure