Seasonal-adjustment Based Feature Selection Method for Predicting Epidemic with Large-scale Search Engine Logs

Thien Q. Tran, Jun Sakuma

Abstract

Search engine logs have a great potential in tracking and predicting outbreaks of infectious disease. More precisely, one can use the search volume of some search terms to predict the infection rate of an infectious disease in nearly real-time. However, conducting accurate and stable prediction of outbreaks using search engine logs is a challenging task due to the following two-way instability characteristics of the search logs. First, the search volume of a search term may change irregularly in the short-term, for example, due to environmental factors such as the amount of media or news. Second, the search volume may also change in the long-term due to the demographic change of the search engine. That is to say, if a model is trained with such search logs with ignoring such characteristic, the resulting prediction would contain serious mispredictions when these changes occur. In this work, we proposed a novel feature selection method to overcome this instability problem. In particular, we employ a seasonal-adjustment method that decomposes each time series into three components: seasonal, trend and irregular component and build prediction models for each component individually. We also carefully design a feature selection method to select proper search terms to predict each component. We conducted comprehensive experiments on ten different kinds of infectious diseases. The experimental results show that the proposed method outperforms all comparative methods in prediction accuracy for seven of ten diseases, in both now-casting and forecasting setting. Also, the proposed method is more successful in selecting search terms that are semantically related to target diseases.

Introduction

Public health officials rely on "now-casting" (predicting the current state) and forecasting to manage epidemics. While search engine logs provide nearly real-time data, using them for prediction is notoriously difficult due to two types of instability. First, short-term irregularity involves sudden spikes in search volume caused by news cycles or media hype rather than actual infections. Second, long-term shifts occur due to evolving user demographics or search engine popularity over years.

Furthermore, many search terms are "seasonally coincident." For example, a term like "junior soccer" might peak at the same time as a winter flu outbreak. Traditional models often pick up these unrelated terms, leading to "overfitting" to the season rather than the actual disease dynamics. This was a primary reason why early efforts like Google Flu Trends eventually failed.

Methodology: Seasonal-Adjustment and Feature Selection

The authors propose a Seasonal-adjustment Based Feature Selection Method using a "divide and conquer" approach to isolate meaningful signals from noise.

Time Series Decomposition

They use a multiplicative model to decompose the logit-transformed infection rates and search term volumes into three distinct components:

  1. Trend ($\hat{T}$): Long-term changes, typically calculated via a 52-week moving average.
  2. Seasonal ($\hat{S}$): The common pattern that repeats every year.
  3. Irregular ($\hat{I}$): Short-term fluctuations that aren't explained by trend or season.
Decomposition of a time series into trend, seasonal, and irregular components.

Feature Selection and Prediction

Instead of ranking search terms by their overall correlation to the disease, they rank them based on how well their individual components (Trend and Irregular) match the corresponding components of the disease data. To ensure semantic relevance, a "seasonal score" is applied to penalize terms that don't share the disease's underlying seasonal rhythm, effectively filtering out noise like the "soccer" example mentioned above.

A "wrapper" approach with Forward Selection and L2-regularized (Ridge) multivariate regression is used to build the model incrementally. Separate models are trained for the trend and irregular components and then recombined with the static seasonal pattern to produce the final epidemic estimate.

Experimental Evaluation

The method was tested using Yahoo! JAPAN search logs (analyzing 2.5 million terms) against ground truth data from the Infectious Diseases Weekly Report (IDWR) in Japan, covering 10 different diseases from 2007 to 2015.

Key Results

  • Superior Accuracy: The proposed method outperformed major baselines (including the original Google Flu Trends method and ElasticNet) in 7 out of 10 diseases.
  • Stability in Forecasting: While baseline models saw their accuracy plummet when trying to predict 1–2 weeks into the future, the proposed method's performance decayed much more slowly.
  • High Relevance Ratio: For Influenza, the model achieved a 100% relevance ratio, selecting only semantically related terms (e.g., "incubation period," "tamiflu").
  • Herpangina Success: For Herpangina, a disease with complex dynamics, the method achieved a correlation of 0.98, significantly higher than baseline methods.

Conclusion

By isolating the "Irregular" component, the model focuses on actual outbreak signals that deviate from the expected yearly cycle. While the method requires a reasonable number of patients and strong seasonality to be most effective, it proves to be a robust, universal framework for epidemic prediction. Future work may involve integrating other data sources, such as social media or weather data, to further improve robustness against long-term trend shifts.

Summary updated

Abstract

Search engine logs have a great potential in tracking and predicting outbreaks of infectious disease. More precisely, one can use the search volume of some search terms to predict the infection rate of an infectious disease in nearly real-time. However, conducting accurate and stable prediction of outbreaks using search engine logs is a challenging task due to the following two-way instability characteristics of the search logs. First, the search volume of a search term may change irregularly in the short-term, for example, due to environmental factors such as the amount of media or news. Second, the search volume may also change in the long-term due to the demographic change of the search engine. That is to say, if a model is trained with such search logs with ignoring such characteristic, the resulting prediction would contain serious mispredictions when these changes occur. In this work, we proposed a novel feature selection method to overcome this instability problem. In particular, we employ a seasonal-adjustment method that decomposes each time series into three components: seasonal, trend and irregular component and build prediction models for each component individually. We also carefully design a feature selection method to select proper search terms to predict each component. We conducted comprehensive experiments on ten different kinds of infectious diseases. The experimental results show that the proposed method outperforms all comparative methods in prediction accuracy for seven of ten diseases, in both now-casting and forecasting setting. Also, the proposed method is more successful in selecting search terms that are semantically related to target diseases.
BibTeX
@inproceedings{tran2019seasonal,
  author = {Thien Q. Tran and Jun Sakuma},
  title = {{Seasonal-adjustment Based Feature Selection Method for Predicting Epidemic with Large-scale Search Engine Logs}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2019},
  url = {https://openreview.net/forum?id=BhgQdDT3bX0},
}
Figure