Introduction
Public health decision-making relies heavily on the integrity of real-time data streams, such as COVID-19 case counts and hospitalization rates. However, these data streams are notoriously noisy, characterized by reporting delays, weekend effects, retrospective corrections, and sudden distribution shifts driven by policy changes or new viral variants. For organizations like the Delphi Group at Carnegie Mellon University, which monitors over 10,000 individual data streams across various geographic scales, manual quality control is a Herculean task.
To address this, researchers developed FlaSH (Flagging Streams in public Health), a scalable framework designed to automatically detect irregularities in high-volume public health data. FlaSH prioritizes "trendline outliers"—the subtle, nuanced irregularities that experts find most helpful to investigate—while maintaining the computational efficiency required for real-time deployment.
The Challenge of Public Health Data
Existing outlier detection frameworks often struggle with the unique characteristics of public health data. Deep learning models, while powerful, are frequently too computationally expensive for real-time use across thousands of streams and require longer historical datasets than are often available during an emerging crisis. Conversely, simple statistical methods often fail to account for systematic "day-of-week" rhythms or regional nuances. FlaSH was designed to bridge this gap by combining robust statistical modeling with a focus on scalability.
The FlaSH Framework
FlaSH formalizes outlier detection as a model-based hypothesis test, operating in three distinct stages:
- Data Processing (S1): To manage non-stationarity (the "waves" of a pandemic), FlaSH utilizes the Pelt Changepoint Algorithm to identify different regimes within the data. It then applies a Poisson regression to remove systematic day-of-week effects and imputes known global outliers to prevent them from biasing the underlying model.
- Prediction (S2): Rather than employing heavy-duty neural networks, FlaSH uses a Linear Autoregressive (AR) model with a lag of seven days. This choice prioritizes interpretability and allows the system to process thousands of streams in minutes rather than hours.
- Statistical Comparison (S3): The core of the framework is a Binomial distribution test statistic used to calculate a p-value. This compares the observed count to the predicted count relative to the region's population. To handle streams with limited history, FlaSH "pools" test statistics across similar geographic regions, creating a robust historical distribution of "normal" performance for that specific area.
Ultimately, FlaSH generates a ranked list of streams where the most recent data point is the most statistically "surprising" relative to the model’s historical accuracy.
Experimental Setup and Datasets
The framework was evaluated using 3,341 COVID-19 case data streams from the JHU CSSE, spanning US counties, states, and territories from May 2020 to May 2022. To establish a ground truth, the researchers collected labels from 13 subject matter experts (statisticians and researchers). These experts ranked data points from five diverse streams—National, Pennsylvania, Arkansas, Queens (NY), and Coamo (PR)—to determine which irregularities actually warranted human intervention.
Key Results
Accuracy and Efficiency
FlaSH demonstrated that scalability does not have to come at the cost of performance. In binary classification tasks (identifying whether a point is an outlier), FlaSH achieved a Balanced Accuracy of 0.73, matching or slightly exceeding DeepLog (0.72), the best-performing deep learning baseline.
Crucially, FlaSH is significantly more efficient. It trained on the full 3,341-stream dataset in just 169 seconds. In contrast, most deep learning baselines failed to complete the task within 24 hours on standard hardware, making them impractical for real-time public health monitoring.
Assistive Value
The framework excels at identifying "hard-to-catch" outliers—subtle irregularities that are unlikely to be noticed by a human reviewer without algorithmic assistance. FlaSH achieved a mean rank of 1.33 for these assistive outliers, consistently placing the most helpful irregularities at the top of the reviewer's list. This significantly outperformed the next best method, DeepLog, which had a mean rank of 2.33.
Conclusion and Future Work
FlaSH demonstrates that for public health stakeholders, interpretability and scalability are essential features. By using simple models like Linear AR, the framework builds trust with users who can easily understand why a specific data point was flagged.
While FlaSH is highly effective at flagging irregularities, it does not yet diagnose the underlying cause (e.g., distinguishing between a reporting error and a genuine biological surge). Future research will focus on adding "explanation" features to help reviewers quickly determine the cause of a flag. Additionally, while currently tuned for COVID-19, the framework's underlying distribution models can be adapted for other public health metrics, such as hospitalizations or mortality rates.
Summary updated