From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsy Narratives

Shuxian Fan, Adam Visokay, Kentaro Hoffman, Stephen Salerno, Liu Li, Jeffrey T. Leek, Tyler H. McCormick

Abstract

In settings where most deaths occur outside the healthcare system, verbal autopsies (VAs) are a common tool to monitor trends in causes of death (COD). VAs are interviews with a surviving caregiver or relative that are used to predict the decedent's COD. Turning VAs into actionable insights for researchers and policymakers requires two steps (i) predicting likely COD using the VA interview and (ii) performing inference with predicted CODs (e.g. modeling the breakdown of causes by demographic factors using a sample of deaths). In this paper, we develop a method for valid inference using outcomes (in our case COD) predicted from free-form text using state-of-the-art NLP techniques. This method, which we call multiPPI++, extends recent work in "prediction-powered inference" to multinomial classification. We leverage a suite of NLP techniques for COD prediction and, through empirical analysis of VA data, demonstrate the effectiveness of our approach in handling transportability issues. multiPPI++ recovers ground truth estimates, regardless of which NLP model produced predictions and regardless of whether they were produced by a more accurate predictor like GPT-4-32k or a less accurate predictor like KNN. Our findings demonstrate the practical importance of inference correction for public health decision-making and suggests that if inference tasks are the end goal, having a small amount of contextually relevant, high quality labeled data is essential regardless of the NLP algorithm.

Introduction

In many regions globally, a significant portion of deaths occur outside formal healthcare systems, leaving a void in medical records. To monitor mortality trends, health workers utilize Verbal Autopsies (VAs)—interviews with the deceased's family. Traditionally, these are structured surveys requiring expert clinician review to determine a Cause of Death (COD). While Natural Language Processing (NLP) offers a way to automate this by predicting COD from free-text narratives, it introduces a significant challenge: statistical bias. If an NLP model is imperfect, using its predictions as "ground truth" in downstream statistical models leads to false conclusions. Furthermore, models often suffer from transportability issues, where a model trained in one geographic or cultural context performs poorly in another.

Methods: The multiPPI++ Framework

To address these challenges, we introduce multiPPI++, a statistical framework that enables valid scientific inference using "noisy" model predictions. The workflow consists of two primary stages:

  1. Prediction: Narratives are classified into five broad COD categories (non-communicable, communicable, external, maternal, and AIDS/TB) using various NLP techniques, ranging from simple "Bag-of-Words" models (SVM, KNN) to BERT and zero-shot GPT-4.
  2. Inference Correction: We extend "Prediction-Powered Inference" (PPI) to the multinomial classification case. The multiPPI++ estimator utilizes a large set of unlabeled narratives (with model predictions) and a small "rectification" set of labeled narratives (with both model predictions and ground truth). By calculating the difference between the model and the truth in the small set, the estimator "de-biases" the statistics calculated on the larger dataset.
Overview of the multiPPI++ framework for correcting model predictions using a labeled rectification set.

Experimental Setup and Datasets

We evaluated our approach using the Population Health Metrics Research Consortium (PHMRC) dataset, which includes 11,979 de-identified free-text narratives of adult deaths collected across six sites in India, the Philippines, Tanzania, and Mexico. This dataset is unique because it contains "gold standard" CODs validated via physical autopsies or hospital records.

To test the framework's robustness against transportability issues, we conducted "leave-one-site-out" experiments. For example, a model might be trained on data from five sites and then used to predict outcomes for a sixth site (e.g., Uttar Pradesh), where only a small fraction of data is labeled for rectification.

Experimental design for cross-site transportability testing and comparison of inference estimators.

Key Results

  • Predictive Performance: GPT-4 achieved the highest raw accuracy (0.75), followed by KNN (0.63) and BERT (0.55). Interestingly, GPT-4's ability to identify "unclassified" narratives highlighted junk data in the original dataset that other models were forced to misclassify.
  • Inference Recovery: In cross-site experiments, a "Naive" approach—treating model predictions as absolute truth—resulted in significantly biased regression coefficients. In contrast, multiPPI++ successfully recovered the ground truth estimates across all models and sites, providing more honest confidence intervals that reflect the inherent uncertainty of predicted data.
  • Accuracy vs. Validity: Our results demonstrate that a highly accurate model can still produce biased population statistics if its errors are systematic. Conversely, even a "weaker" model like KNN can yield valid scientific insights when the multiPPI++ correction is applied.

Discussion and Limitations

While multiPPI++ provides a powerful tool for public health researchers, it still requires a small amount of high-quality, contextually relevant labeled data for the rectification step. Future work should address language barriers, as this study focused on English narratives, whereas VAs are most critical in non-English speaking regions. The findings suggest that for many inference tasks, combining a cost-effective NLP model with a small labeled correction set is more efficient than relying solely on expensive, state-of-the-art LLMs.

Summary updated

Abstract

In settings where most deaths occur outside the healthcare system, verbal autopsies (VAs) are a common tool to monitor trends in causes of death (COD). VAs are interviews with a surviving caregiver or relative that are used to predict the decedent's COD. Turning VAs into actionable insights for researchers and policymakers requires two steps (i) predicting likely COD using the VA interview and (ii) performing inference with predicted CODs (e.g. modeling the breakdown of causes by demographic factors using a sample of deaths). In this paper, we develop a method for valid inference using outcomes (in our case COD) predicted from free-form text using state-of-the-art NLP techniques. This method, which we call multiPPI++, extends recent work in "prediction-powered inference" to multinomial classification. We leverage a suite of NLP techniques for COD prediction and, through empirical analysis of VA data, demonstrate the effectiveness of our approach in handling transportability issues. multiPPI++ recovers ground truth estimates, regardless of which NLP model produced predictions and regardless of whether they were produced by a more accurate predictor like GPT-4-32k or a less accurate predictor like KNN. Our findings demonstrate the practical importance of inference correction for public health decision-making and suggests that if inference tasks are the end goal, having a small amount of contextually relevant, high quality labeled data is essential regardless of the NLP algorithm.
BibTeX
@misc{fan2024from,
  author = {Shuxian Fan and Adam Visokay and Kentaro Hoffman and Stephen Salerno and Liu Li and Jeffrey T. Leek and Tyler H. McCormick},
  title = {{From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsy Narratives}},
  year = {2024},
  eprint = {2404.02438},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.02438},
}

Loading related content...

Figure