Optimizing Global Influenza Surveillance for Locations with Deficient Data (Student Abstract)

Songwei Shan, Qi Tan, Yiu Chung Lau, Zhanwei Du, Eric H. Y. Lau, Peng Wu, Benjamin J. CowlingAAAI 2022 · DOI

Abstract

Effective global health response is often hindered by sparse or inconsistent epidemiological data. This study presents a novel surveillance system that combines a sparse linear model with a temporal weighting strategy to track influenza in data-deficient regions. By leveraging $L_1$ regularization and a logarithmic decay function to account for concept drift, the proposed Multi-Source Predictive Model outperforms complex deep learning architectures, such as LSTMs, in predicting infection trends across 132 countries.

Introduction: The Challenge of Deficient Data

High-quality data is the cornerstone of effective global health responses. While the World Health Organization (WHO) maintains the FluMART platform for collecting global influenza data, significant reporting gaps persist. Between 2010 and 2020, approximately 78% of participating countries provided complete data for less than half of the weeks in that decade.

This "deficient data" problem arises from late reporting starts, resource constraints, and "concept drift"—changes in clinical criteria or reporting standards over time. Traditional predictive models often struggle with these inconsistencies, leading to inaccurate epidemic status assessments and inefficient resource allocation.

Visual representation of data sparsity and reporting inconsistencies in Armenia and Austria.

Methodology

The researchers developed a Multi-Source Predictive Model that structures global influenza data as a three-dimensional tensor (Countries × Attributes × Time). The model is built on two primary technical pillars:

  1. Robust Feature Selection via Sparsity: To mitigate noise, the model employs $L_1$ regularization (Lasso). This technique selects a small, highly informative subset of reliable indicators. The linear nature of the model ensures computational efficiency and allows for modeling "cross-region influence," where outbreaks in one country can serve as signals for neighboring regions.
  2. Temporal Weighting for Concept Drift: To address evolving reporting standards, a weighting function ($u_t$) was introduced. This function applies a logarithmic decay, assigning higher importance to recent data. This ensures the model remains sensitive to current reporting practices rather than being biased by historical data that may no longer be relevant.

Results and Evaluation

The model was trained on FluMART data from 132 countries spanning 2009 to 2020, targeting the number of Influenza A positive samples. Performance was benchmarked against three standard models using Root Mean Square Error (RMSE).

Method RMSE
Multilayer Perceptron (MLP) 1.1632e+04
Linear (without selection) 9.3410e+03
Long Short-Term Memory (LSTM) 7.8207e+03
Proposed Method 6.5383e+03

The proposed method achieved the lowest RMSE, demonstrating superior robustness in the face of missing data and noise. Qualitative analysis further supports these findings; for instance, in Turkey, the model's predictions closely aligned with the ground-truth spikes in Influenza A cases.

Prediction results for Turkey, demonstrating the model's ability to accurately track actual infection spikes over time.

Discussion and Conclusion

A key takeaway from this research is that simplicity can outperform complexity in noisy environments. While deep learning models like LSTMs are powerful, they are prone to overfitting when faced with the significant "gaps" found in global health data. A sparse linear approach, combined with strategic temporal weighting, provides a more reliable framework for real-world surveillance.

Future work will focus on expanding the model's reach to environments with even fewer consistent indicators and applying the framework to other infectious diseases, such as COVID-19 and RSV, which face similar reporting challenges.

Summary updated

BibTeX
@inproceedings{shan2022optimizing,
  author = {Songwei Shan and Qi Tan and Yiu Chung Lau and Zhanwei Du and Eric H. Y. Lau and Peng Wu and Benjamin J. Cowling},
  title = {{Optimizing Global Influenza Surveillance for Locations with Deficient Data (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21659},
}
Figure