Abstract
Search engine logs have a great potential in tracking and predicting outbreaks of infectious disease. More precisely, one can use the search volume of some search terms to predict the infection rate of an infectious disease in nearly real-time. However, conducting accurate and stable prediction of outbreaks using search engine logs is a challenging task due to the following two-way instability characteristics of the search logs. First, the search volume of a search term may change irregularly in the short-term, for example, due to environmental factors such as the amount of media or news. Second, the search volume may also change in the long-term due to the demographic change of the search engine. In this work, we proposed a novel feature selection method to overcome this instability problem. In particular, we employ a seasonal-adjustment method that decomposes each time series into three components: seasonal, trend and irregular component and build prediction models for each component individually. We also carefully design a feature selection method to select proper search terms to predict each component. We conducted comprehensive experiments on ten different kinds of infectious diseases. The experimental results show that the proposed method outperforms all comparative methods in prediction accuracy for seven of ten diseases, in both now-casting and forecasting setting. Also, the proposed method is more successful in selecting search terms that are semantically related to target diseases.
Introduction
Using search engine logs to track diseases (now-casting) and predict outbreaks (forecasting) is a significant goal for public health, as search data is available in near real-time compared to delayed hospital reports. However, historical attempts like "Google Flu Trends" have famously failed by overestimating prevalence. The primary challenges identified by researchers are "instability" factors: short-term spikes from news cycles, long-term demographic shifts in search engine usage, and "spurious seasonality" where unrelated terms (e.g., "hot chocolate") correlate with seasonal diseases (e.g., flu) simply due to timing.
Methodology
The proposed framework addresses these issues through a Seasonal-adjustment Based Feature Selection approach. The core innovation lies in decomposing the high-dimensional search data into interpretable components to filter out noise.
Decomposition and Modeling
The authors apply a multiplicative model to decompose the logit-transformed time series of both the infection rate ($Y$) and every candidate search term ($X^k$) into three distinct parts:
- Trend ($T$): Captures long-term changes, such as demographic shifts.
- Seasonal ($S$): Represents the repeating annual cycle.
- Irregular ($I$): Contains short-term fluctuations, which often represent the actual "outbreak" signal.
Instead of a single monolithic model, individual models are built for the Trend and Irregular components. The Seasonal component is assumed to be stable and is modeled using historical averages. This separation allows the model to handle long-term demographic shifts without letting them corrupt the detection of sudden outbreaks.
Feature Selection
A novel scoring function ranks search terms by multiplying their correlation with the target's trend/irregular component by a Seasonal Score. This effectively penalizes terms that do not share the disease's seasonal rhythm, filtering out irrelevant terms early. A forward selection process with an L2-regularized multivariate linear regression model then identifies the optimal subset of search terms for each component.
Experimental Results
The method was tested using the Infectious Diseases Weekly Report (IDWR) from Japan (10 diseases) and Yahoo! JAPAN search logs (2.5 million terms).
- Accuracy and Stability: The proposed method outperformed baselines like ElasticNet and Google Flu Trends in 7 out of 10 diseases. It showed particular strength in forecasting (1-2 weeks ahead), where baseline accuracy typically drops sharply.
- Interpretability: For Influenza, the model's "relevance ratio" reached nearly 100%, selecting terms like "fever medicine" or "incubation period" rather than unrelated seasonal terms.
- Performance: In specific cases like herpangina, the model achieved a correlation of 0.98, a 12% improvement over the next best method.
Conclusion
By decomposing search logs into trend, seasonal, and irregular components, this method successfully filters out "spurious correlations" and handles long-term demographic shifts. While it requires sufficient "crowd signal" to be effective, it provides a more robust and interpretable way to leverage search data for public health. Future work may involve integrating non-linear models like Neural Networks into this decomposition framework to further enhance predictive power.
Summary updated