TL;DR
This paper introduces a novel framework for evaluating the suitability of influenza surveillance systems for AI/ML-driven short-term forecasting. By identifying key attributes like timeliness and specificity, the study pinpoints the most promising systems in New Zealand for improving influenza prediction and public health response.
What Problem Does This Paper Address?
Influenza remains a significant global health threat, causing seasonal epidemics and occasional pandemics that strain healthcare systems and lead to considerable morbidity and mortality. Effective public health interventions rely on accurate and timely monitoring of influenza activity. While artificial intelligence (AI) and machine learning (ML) offer powerful tools for forecasting disease spread, their effectiveness hinges on the quality and suitability of the data they consume. Existing surveillance system evaluation frameworks, however, were not specifically designed to assess their utility for AI/ML applications. This paper addresses this gap by proposing a method to evaluate and compare influenza surveillance systems based on their potential to support AI/ML-based short-term forecasting, particularly in a high-income country context like New Zealand.
Methods
The study employed a two-phase approach.
Phase 1: System Characterization: This involved a comprehensive review of government reports, official websites, and academic literature to gather detailed information on existing influenza surveillance systems in New Zealand.
Phase 2: System Evaluation: A framework was developed to evaluate these systems against eight key attributes: timeliness, sensitivity, specificity, representativeness, coverage, robustness, completeness, and historical data. These attributes were selected based on their relevance to AI/ML requirements, their consistent use in existing frameworks, and expert knowledge. A five-level ranking system was used, and weighted scores were calculated to assess each system's "Usefulness for training" (prioritizing historical data, accuracy, and completeness) and "Usefulness for short-term forecasting" (prioritizing timeliness, robustness, and sensitivity).
Datasets
The study evaluated existing influenza surveillance systems in New Zealand. Specific systems mentioned include:
- Community-based: HealthStat, Healthline, FluTracking, Southern Hemisphere Influenza and Vaccine Effectiveness Research and Surveillance (SHIVERS), Google Flu Trends.
- Hospital-based: Severe Acute Respiratory Infection (SARI), National Minimum Data Set (NMDS).
- Laboratory-based: Laboratory-Based Surveillance (The National Influenza Center).
- Mortality: Mortality Datasets from the Ministry of Health.
The paper does not introduce new benchmark datasets but rather evaluates the suitability of existing data sources for AI/ML applications.
Key Insights and Ideas
- No single system is perfect: Different surveillance systems have distinct strengths and weaknesses, making a multi-system approach crucial for robust AI/ML forecasting.
- Timeliness is paramount for forecasting: Systems that provide near real-time data are essential for short-term predictions, even if they sacrifice some specificity.
- Historical data is key for training: For ML models to learn effectively, access to comprehensive historical data is vital, even if that data isn't as timely for immediate forecasting.
- Laboratory confirmation boosts accuracy: Systems that incorporate laboratory-confirmed influenza cases (like SHIVERS and SARI) generally score higher in sensitivity and specificity, making them valuable for both training and forecasting.
- Bridging community and hospital data is important: Laboratory-based surveillance plays a critical role in connecting different data streams and improving the overall picture.
Key Results
The study identified SHIVERS (community-based) and SARI (hospital-based) as the most useful systems for AI/ML applications in New Zealand.
- SHIVERS achieved the highest scores for both training (4.09) and short-term forecasting (4.20) in community settings, largely due to its influenza virus-specific, laboratory-confirmed data.
- SARI also performed strongly, with the highest short-term forecasting score (4.05) and high training performance (4.10) due to its longitudinal data, sensitivity, and specificity.
- The National Minimum Dataset (NMDS) showed strong training potential (4.33) due to its extensive historical data but was limited in short-term forecasting due to timeliness constraints.
- Laboratory-based surveillance was found to be useful for bridging community and hospital data, offering moderate utility for both training and forecasting.
- Google Flu Trends, while real-time, was found to have low specificity and was eventually discontinued, highlighting the risks of relying on less robust data sources.
Limitations and Future Work
Limitations:
- The subjective nature of attribute rankings, though validated by experts, may limit the generalizability of the conclusions.
- Sensitivity and specificity were inferred from proxy indicators rather than direct measurement.
- The study raises hypotheses about multisource datasets that require further practical investigation.
Future Work:
- Develop a training dataset aligned with different levels of influenza severity, incorporating well-suited data sources.
- Explore the application of transfer learning within a multivariate framework to integrate time-series data across severity levels.
- Investigate forecasting using diverse datasets and specific models to capture distinct features of influenza activity.
- Further research is needed to assess the feasibility and implications of integrating syndromic and etiological data from different severity levels.
- Consider improving the timeliness of systems like NMDS to enhance their short-term forecasting capabilities.
Summary updated