Abstract
Influenza is an acute respiratory seasonal disease that affects millions of people worldwide and causes thousands of deaths in Europe alone. Being able to estimate in a fast and reliable way the impact of an illness on a given country is essential to plan and organize effective countermeasures, which is now possible by leveraging unconventional data sources like web searches and visits. In this study, we show the feasibility of exploiting information about Wikipedia's page views of a selected group of articles and machine learning models to obtain accurate estimates of influenza-like illnesses incidence in four European countries: Italy, Germany, Belgium, and the Netherlands. We propose a novel language-agnostic method, based on two algorithms, Personalized PageRank and CycleRank, to automatically select the most relevant Wikipedia pages to be monitored without the need for expert supervision. We then show how our model is able to reach state-of-the-art results by comparing it with previous solutions.
Introduction
Monitoring seasonal influenza is a critical task for public health organizations like the European Centre for Disease Prevention and Control (ECDC). Traditional surveillance relies on networks of sentinel physicians who report influenza-like illness (ILI) cases. However, this data often has a one-to-two-week reporting lag. In epidemiology, this delay can hinder timely preventive measures.
While previous research has explored "nowcasting" ILI using Wikipedia pageviews, these methods often relied on expert-curated lists of articles. Such manual approaches are difficult to scale across different languages and countries. This study introduces a language-agnostic, automated method for feature selection that leverages the internal link structure of Wikipedia to identify relevant articles.
Methodology: Automated Feature Selection
The core innovation of this research is treating Wikipedia as a directed graph where articles are nodes and hyperlinks are edges. The researchers compared three selection methods:
- Manual Categories: A baseline using pages within predefined medical categories.
- Personalized PageRank (PPageRank): Ranks pages based on the probability of reaching them via random walks starting from the "Influenza" article.
- CycleRank: A novel algorithm focusing on "circular walks" (paths starting and ending at the source node). This method effectively filters out "hubs"—generic pages with many incoming links that lack specific contextual relevance to the disease.
The selected pageview data is then fed into a linear model regularized with LASSO. LASSO performs automatic feature selection by shrinking the weights of less predictive variables to zero, ensuring the final model uses only the most relevant Wikipedia pages.
Results
The automated models consistently matched or outperformed manual curation across all studied countries. CycleRank, in particular, proved superior by surfacing specific symptom-related pages while avoiding the "hub" problem associated with standard PageRank.
Key performance metrics included:
- High Correlation: Pearson Correlation Coefficients (PCC) often exceeded 0.90 for Italy and 0.85 for Germany.
- Peak Prediction: The models successfully predicted the peak week of the flu season within a ±2 week window, providing a significant lead over traditional reporting.
- Language Agnosticism: Because the method relies on link structure rather than semantic analysis, it can be deployed in any language given only the seed term for "Influenza."
Discussion and Future Work
Despite the high accuracy, the models occasionally exhibited "spikes" unrelated to actual ILI incidence. These are typically attributed to "media effects," where news stories drive pageviews regardless of actual illness.
Future research will focus on:
- Anomaly Detection: Developing filters to distinguish between organic health-seeking behavior and news-driven spikes.
- Advanced Modeling: Exploring Poisson or Negative Binomial regression models, which are better suited for epidemiological count data.
- NLP Integration: Using Natural Language Processing to analyze the content of trending news and its relevance to local disease prevalence.
Summary updated