AI-Accelerated Design of Evidence Synthesis for Global Development

Kristen M. Edwards, Binyang Song, Jaron Porciello, Mark Engelbert, Carolyn Huang, Faez Ahmed2023 · DOI

TL;DR

This paper introduces ADVISE, an AI-accelerated system that uses BERT and active learning to significantly speed up the creation of evidence synthesis products (like Evidence Gap Maps) for global development. By integrating AI into human teams, they demonstrate a substantial reduction in human screening effort, making evidence-based policymaking more efficient.

What Problem Does This Paper Address?

Decision-makers in global development rely on evidence synthesis to inform policies, but the sheer volume of research literature makes this process incredibly time-consuming and resource-intensive. Manually sifting through thousands of documents to identify relevant studies for products like Evidence Gap Maps (EGMs) can take months. This delay hinders timely, evidence-based decision-making. The paper aims to bridge this gap by accelerating the initial title and abstract screening phase of EGM creation.

Methods

The core of ADVISE is a BERT-based AI agent designed for title and abstract screening. BERT, a powerful transformer-based language model, is fine-tuned for this task. To further enhance efficiency, the researchers incorporate active learning (AL). AL allows the AI to intelligently select which documents humans should review, reducing the need for exhaustive manual screening. They explore three AL sampling strategies:

  • Random Sampling: No AI guidance, serving as a baseline.
  • Least Confidence (LC) Sampling: The AI prioritizes documents it's most uncertain about.
  • Highest Priority (HP) Sampling: The AI prioritizes documents it predicts are most likely to be relevant.

The system operates within a human-AI hybrid team workflow, where humans define criteria, the AI trains on initial human-labeled data, predicts relevance scores, and then suggests papers for human review. This process iterates, allowing the AI to learn and improve over time.

Datasets

The study utilizes two main types of datasets:

  • Simulated EGM Design Dataset: A fully labeled dataset of 68,539 documents from 3ie's Development Evidence Portal (DEP). This dataset is used for retrospective analysis and to evaluate different models and training sizes. It's imbalanced, with 5,281 included papers and 63,258 excluded papers.
  • Deployed EGM Design Datasets: For the real-world application, they created three EGMs (Agriculture, Nutrition, and Resilience) using initial literature searches that yielded large datasets (221k for Agriculture, 117k for Nutrition, and 60k for Resilience). These datasets were partially labeled through the human-AI screening process.

Key Insights and Ideas

  • BERT's Superiority: BERT-based models significantly outperform traditional SVM models in accuracy for classifying relevant documents in the EGM context.
  • Active Learning's Impact: Active learning, particularly HP sampling, dramatically reduces human effort beyond what BERT alone can achieve. The AI's ability to intelligently guide human review is crucial.
  • Human-AI Collaboration is Key: The paper emphasizes that AI doesn't replace humans but augments their capabilities. The "hybrid" nature is essential for effective evidence synthesis.
  • Training Data Optimization: There's a sweet spot for training data size. Too little, and the AI is ineffective; too much, and the human effort to label it outweighs the benefits. Around 5,000 documents proved optimal in their simulated experiments.
  • The "When to Stop" Dilemma: A practical challenge in deployed EGM design is knowing when to stop the iterative AI training and human screening process. This is framed as a business decision balancing resource cost with improved model performance.

Key Results

  • Human Effort Reduction:
    • Compared to no AI assistance, the BERT-based AI agent reduced human screening effort by 68.5% for identifying 80% of relevant documents.
  • Compared to an SVM-based agent (industry standard), BERT reduced effort by 16.8%.
  • With the Highest Priority (HP) active learning strategy, human effort was further reduced by 78.3% compared to no AI assistance.
  • Accuracy: BERT models consistently showed higher classification accuracy than SVM models across the three deployed EGMs.
  • Active Learning Efficiency: Active learning strategies (LC and HP) saved an additional ~30% of human effort compared to BERT without active learning.
  • Practical Validation: The AL-enhanced human-AI hybrid workflow was successfully applied to design three EGMs for USAID.

Limitations and Future Work

  • Full-text Screening Automation: The current work focuses on title and abstract screening. Automating full-text screening is a logical next step but faces challenges with document access (copyright) and requires training new models, potentially with LLMs.
  • Counterfactual Analysis: Due to the nature of deployed EGM design, true counterfactual analyses (comparing EGM creation with and without AI assistance from scratch) were not fully feasible. Future work could address this by creating EGMs multiple times using different techniques.
  • "When to Stop" Criteria: Developing more robust, data-driven criteria for determining when to stop the AI training and human screening loop is an area for future research.
  • LLM Integration: The authors suggest that Large Language Models (LLMs) like ChatGPT could play a significant role in future EGM design, potentially assisting with tasks like identifying intervention-outcome relationships or generating summaries, even without domain-specific training data.

Summary updated

Abstract

Abstract When designing evidence-based policies and programs, decision-makers must distill key information from a vast and rapidly growing literature base. Identifying relevant literature from raw search results is time and resource intensive, and is often done by manual screening. In this study, we develop an AI agent based on a bidirectional encoder representations from transformers (BERT) model and incorporate it into a human team designing an evidence synthesis product for global development. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. To further improve team efficiency, we enhance the human-AI hybrid team through active learning (AL). Specifically, we explore different sampling strategies, including random sampling, least confidence (LC) sampling, and highest priority (HP) sampling, to study their influence on the collaborative screening process. Results show that incorporating the BERT-based AI agent into the human team can reduce the human screening effort by 68.5% compared to the case of no AI assistance and by 16.8% compared to the industry standard of using an n-gram language model to encode texts and a support vector machine (SVM)-based classifier for identifying 80% of all relevant documents. When we apply the HP sampling strategy for AL, the human screening effort can be reduced even more to 78.3% for identifying 80% of all relevant documents compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming workflow in the design process of three evidence gap maps (EGMs) for USAID and find it to be highly effective. These findings demonstrate how AI can accelerate the development of evidence synthesis products and promote timely evidence-based decision making in global development in a human-AI hybrid teaming context.
BibTeX
@misc{edwards2023ai,
  author = {Kristen M. Edwards and Binyang Song and Jaron Porciello and Mark Engelbert and Carolyn Huang and Faez Ahmed},
  title = {{AI-Accelerated Design of Evidence Synthesis for Global Development}},
  year = {2023},
  doi = {10.1115/detc2023-116962},
}
Figure