Eliciting Disease Data from Wikipedia Articles

Geoffrey Fairchild, Lalindra De Silva, Sara Y. Del Valle, Alberto M. SegrearXiv 2015 · arXiv · OpenReview

TL;DR

This paper proposes using Wikipedia articles as a novel, community-driven source for real-time disease surveillance data. By employing named-entity recognition to extract case counts and analyzing tabular data, the authors demonstrate Wikipedia's potential to create an open-source system for detecting, monitoring, and archiving emerging disease information, overcoming limitations of traditional surveillance.

What Problem Does This Paper Address?

Traditional disease surveillance systems, often reliant on patient visits or lab records, suffer from significant reporting lags (up to weeks) and can be unreliable in regions with limited infrastructure. While internet-based systems like HealthMap have emerged to provide near real-time data, they often lack transparency due to closed-source code and proprietary data, hindering community contribution and replication. This paper addresses the need for an open, accessible, and timely source of disease data by exploring the untapped potential of Wikipedia's vast and constantly updated content.

Methods

The authors employ two primary methods to extract disease-related information from Wikipedia:

  1. Named-Entity Recognition (NER): They train a custom NER model, specifically using Stanford's NER implementation, to identify and categorize key disease metrics within article narratives. The model is trained to recognize three specific entity types: DEATHS, INFECTIONS, and HOSPITALIZATIONS. This approach moves beyond simple keyword matching by leveraging sequence labeling to capture contextual information and handle variations in phrasing.
  2. Tabular Data Analysis: The researchers analyze the tabular data present in Wikipedia articles, focusing on time-series data. For the 2014 West African Ebola epidemic article, they extract case and death counts from tables, noting the frequency of updates and the inclusion of references for data validation.

Datasets

  • Training Data: The NER model was trained on a dataset derived from 14 Wikipedia articles covering various disease outbreaks and epidemics. This dataset comprised approximately 55,000 tokens.
  • Evaluation Data: The tabular data analysis used the 2014 West African Ebola virus disease epidemic Wikipedia article. Time series data from this article were compared against ground truth data sourced from official WHO data and reports.

Key Insights and Ideas

  • Wikipedia as a Real-time Surveillance Source: The study highlights that Wikipedia articles, particularly those on unfolding epidemics, are updated frequently and contain detailed information that can be extracted for surveillance purposes.
  • NER's Efficacy for Disease Metrics: Training a specialized NER model proved effective in extracting crucial disease counts (cases, deaths, hospitalizations) from unstructured text, achieving a respectable F1 score. This demonstrates that even complex, varied language can be parsed for specific data points.
  • Tabular Data's Accuracy: The analysis of tabular data from the Ebola epidemic article showed that Wikipedia's time-series data closely align with official ground truth data, with deviations typically within 1-2% of the reported figures.
  • Community-Driven Potential: The authors strongly advocate for Wikipedia as the foundation for a community-driven, open-source surveillance system, leveraging its collaborative nature and readily available data.

Key Results

  • NER Performance: The trained named-entity recognizer achieved an F1 score of 0.753 for identifying disease-related entities (DEATHS, INFECTIONS, HOSPITALIZATIONS). The best performance was observed with a maxNGramLeng parameter of 4.
  • Tabular Data Accuracy: The root-mean-square error (RMSE) between Wikipedia's time-series data and WHO-based ground truth data for the Ebola epidemic was generally low. In the worst case, the average deviation was approximately 19 cases and 12 deaths, representing only 1-2% of the total reported figures for the affected countries.
  • Data Granularity: Wikipedia's tabular data for the Ebola epidemic provided time granularity that was irregular but generally updated every 2-5 days.

Limitations and Future Work

  • Training Data Size: The authors acknowledge that more training data is needed to improve the NER model's precision and recall for an operational system.
  • NER Correctness and Timeliness: While the study demonstrates the methods for recognizing entities, a thorough study of the correctness and timeliness of the extracted data is still required.
  • Tabular Data Scope: The analysis of tabular data was limited to a single article (Ebola epidemic). A more extensive study across multiple articles is necessary.
  • Language Support: The current work focuses on English Wikipedia. Future work should explore the quality of data in non-English Wikipedias, as NER models can be adapted to other languages.
  • Edit Summaries: The study did not utilize edit summaries, which could provide valuable information about article vandalism, corrections, and the reasons behind edits, potentially improving model performance.
  • Ground Truth Staticity: The static nature of the ground truth time series used for RMSE calculation might inflate errors if it contains corrections for earlier reporting mistakes.

Summary updated

Abstract

Traditional disease surveillance systems suffer from several disadvantages, including reporting lags and antiquated technology, that have caused a movement towards internet-based disease surveillance systems. Internet systems are particularly attractive for disease outbreaks because they can provide data in near real-time and can be verified by individuals around the globe. However, most existing systems have focused on disease monitoring and do not provide a data repository for policy makers or researchers. In order to fill this gap, we analyzed Wikipedia article content. We demonstrate how a named-entity recognizer can be trained to tag case counts, death counts, and hospitalization counts in the article narrative that achieves an F1 score of 0.753. We also show, using the 2014 West African Ebola virus disease epidemic article as a case study, that there are detailed time series data that are consistently updated that closely align with ground truth data. We argue that Wikipedia can be used to create the first community-driven open-source emerging disease detection, monitoring, and repository system.
BibTeX
@misc{fairchild2015eliciting,
  author = {Geoffrey Fairchild and Lalindra De Silva and Sara Y. Del Valle and Alberto M. Segre},
  title = {{Eliciting Disease Data from Wikipedia Articles}},
  year = {2015},
  eprint = {1504.00657},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1504.00657},
}
Figure