An Interpretable Predictive Model of Vaccine Utilization for Tanzania

Ramkumar Hariharan, Johnna Sundberg, Giacomo Gallino, Ashley Schmidt, Drew Arenth, Suvrit Sra, Benjamin FelsFrontiers Artif. Intell 2020 · DOI

TL;DR

This paper introduces a novel machine learning approach to predict vaccine utilization at individual health facilities in Tanzania, achieving significantly higher accuracy than existing methods. By leveraging rich, temporally and regionally relevant data, the model not only forecasts demand but also provides interpretable insights into the factors driving vaccine uptake, paving the way for more efficient vaccine distribution and reduced wastage.

What Problem Does This Paper Address?

Accurate forecasting of vaccine utilization is crucial for optimizing vaccine stock management in healthcare facilities, especially in low- and middle-income countries. Current methods often rely on outdated census data and simplistic forecasting models that offer little insight into the complex factors influencing demand. This can lead to either under-stocking, resulting in reduced vaccination coverage, or over-stocking, causing vaccine wastage. The challenge lies in developing a predictive system that is both accurate and interpretable, capable of handling the multivariate nature of vaccine utilization and providing actionable insights for public health programs.

Methods

The authors developed a machine learning model using a Random Forest Regressor (RFR). This approach is a significant departure from traditional methods by employing a powerful, multivariate technique. Key methodological aspects include:

  • Novel Data Integration: The model utilizes not only vaccine utilization data but also a comprehensive set of features related to health facilities (e.g., type, ownership, geo-coordinates, altitude) and catchment populations (e.g., regional population, births).
  • Temporal and Regional Relevance: The model leverages bi-weekly utilization data, which is more granular and recent than traditional census data, and is trained on data from specific regions in Tanzania.
  • Feature Engineering: The datetime feature from immunization data was ingeniously split into 12 separate temporal features (e.g., day of week, month start/end) using the fastai library. A rolling 3-month average of utilization was also incorporated as a feature.
  • Model Selection: A range of common machine learning algorithms were evaluated, with RFR demonstrating superior performance and interpretability compared to others like gradient boosting, KNN, and neural networks.
  • Interpretability: The RFR's "mean decrease accuracy" method was used to identify feature importances, providing insights into what drives vaccine utilization predictions.

Datasets

The study utilized several datasets:

  • Vaccine Utilization Data: Daily vaccine utilization data from 710 health facilities across three regions in Tanzania (Arusha, Tanga, and Kilimanjaro). This data was aggregated into bi-weekly utilization for forecasting.
  • Tanzania Health Facility Registry (THFR): Provided data on health facility characteristics.
  • Tanzania National Bureau of Statistics (TNBS): Provided population-related data.
  • GPS Visualizer: Used to obtain elevation data for health facilities.

The paper does not explicitly state the creation of new benchmark datasets but rather the novel integration and augmentation of existing data sources.

Key Insights and Ideas

  • The Power of Granular Data: Bi-weekly, facility-level vaccine utilization data, combined with rich contextual features, is far more predictive than traditional, aggregated, or outdated data sources.
  • Multivariate Nature of Demand: Vaccine utilization is influenced by a complex interplay of factors beyond simple historical trends, including facility characteristics and local demographics.
  • Interpretability is Key: The RFR model not only predicts but also explains why it makes certain predictions, offering actionable insights for public health managers. This is a significant advantage over black-box models.
  • Rolling Averages are Strong Predictors: The three-month rolling average of vaccine utilization emerged as the most significant feature in Model I, highlighting the importance of recent trends.
  • Facility Characteristics Matter: Features like geo-coordinates, facility type, and ownership implicitly capture demand characteristics of the catchment population.

Key Results

  • High Accuracy: The RFR model achieved a forecasting fraction error (F.E.) of less than 0.2 for approximately 45% of regional health facilities in the analyzed regions.
  • Significant Improvement: The RFR model's average forecasting fraction error was almost 18 times less than the existing system (Table 3).
  • Specific Vaccine Performance: The model performed best for rotavirus vaccine utilization, with nearly 50% of facilities achieving an F.E. < 0.2.
  • Feature Importance: For Model I, the three-month rolling average was the most important feature. For Model II (which excluded the rolling average), "Vaccine type" was the most important feature.
  • Model II Generalizability: Model II, which uses features derivable from publicly available resources, achieved an F.E. < 0.2 for 15-20% of facilities, demonstrating its potential for broader application.

Limitations and Future Work

  • Data Availability: The study acknowledges that vaccine campaign data was not available and suggests its inclusion in future work could lead to even more accurate models.
  • Generalizability of Model II: While Model II is more generalizable, its predictive performance is slightly lower than Model I. Future work could focus on improving its accuracy while retaining its broad applicability.
  • External Factors: The model does not explicitly account for external shocks or interventions (e.g., sudden disease outbreaks, large-scale vaccination campaigns) that could significantly impact utilization.
  • Future Directions: The authors propose that this work serves as a starting point for reimagining predictive health systems, emphasizing the potential of AI and big data for improving global health outcomes. They also suggest adapting the model to other countries and regions.

Summary updated

BibTeX
@article{hariharan2020interpretable,
  author = {Ramkumar Hariharan and Johnna Sundberg and Giacomo Gallino and Ashley Schmidt and Drew Arenth and Suvrit Sra and Benjamin Fels},
  title = {{An Interpretable Predictive Model of Vaccine Utilization for Tanzania}},
  journal = {Frontiers Artif. Intell.},
  year = {2020},
  doi = {10.3389/frai.2020.559617},
}
Figure