Abstract
In the current technological era, huge amounts of big data are generated and collected from a wide variety of rich data sources. These big data can be of different levels of veracity in the sense that some of them are precise while some others are imprecise and uncertain. Embedded in these big data are useful information and valuable knowledge to be discovered. An example of these big data is healthcare and epidemiological data such as data related to patients who suffered from epidemic diseases like the coronavirus disease 2019 (COVID-19). Knowledge discovered from these epidemiological data—via data science techniques such as machine learning, data mining, and online analytical processing (OLAP)—helps researchers, epidemiologists and policy makers to get a better understanding of the disease, which may inspire them to come up ways to detect, control and combat the disease. In this paper, we present a machine learning and big data analytic tool for processing and analyzing COVID-19 epidemiological data. Specifically, the tool makes good use of taxonomy and OLAP to generalize some specific attributes into some generalized attributes for effective big data analytics. Instead of ignoring unknown or unstated values of some attributes, the tool provides users with flexibility of including or excluding these values, depending on their preference and applications. Moreover, the tool discovers frequent patterns and their related patterns, which help reveal some useful knowledge such as absolute and relative frequency of the patterns. Furthermore, the tool learns from the patterns discovered from historical data and predicts useful information such as clinical outcomes for future data. As such, the tool helps users to get a better understanding of information about the confirmed cases of COVID-19. Although this tool is designed for machine learning and analytics of big epidemiological data, it would be applicable to machine learning and analytics of big data in many other real-life applications and services.
Introduction: The Challenge of High-Volume, Low-Veracity Data
During the height of the COVID-19 pandemic, the global community was inundated with massive amounts of epidemiological data. However, much of this information was "dirty"—incomplete, highly specific (raising privacy concerns), or updated so rapidly that traditional analysis could not keep pace.
Most existing public dashboards at the time focused on simple, one-dimensional metrics, such as daily case counts or total cumulative deaths by region. While these metrics provided a necessary snapshot of the pandemic's scale, they failed to reveal the complex, multi-dimensional relationships between factors like age, occupation, transmission method, and clinical outcome.
The primary challenge addressed in this research is turning this high-volume, low-veracity data into actionable knowledge. By moving beyond basic tallies, researchers and policy makers can gain deeper insights into how the disease spreads and affects different populations.
Methodology: Integrating OLAP and Machine Learning
The proposed tool utilizes a multi-stage pipeline that integrates traditional database techniques with predictive modeling:
- Taxonomy-Based Generalization (OLAP): To reduce dimensionality and protect privacy, the tool employs hierarchical taxonomies. For instance, specific ages are binned into groups (e.g., "20s," "30s"), and specific locations are rolled up into broader regions. This "densification" makes the data more suitable for pattern mining.
- Explicit NULL Handling: Unlike many machine learning models that discard records with missing values, this tool treats "NULL" or "unstated" as a distinct, informative category. This prevents the loss of significant data and reflects real-world reporting gaps.
- Frequent Pattern Mining: The tool identifies "singleton" patterns (e.g., {community exposure}) and "non-singleton" patterns (e.g., {female, community exposure, recovered}).
- Associative Classification: This serves as the bridge to machine learning. Discovered patterns are transformed into association rules (e.g., "If A and B, then C"), which are then used to predict clinical outcomes for new data.
Data and Experimental Evaluation
The tool was evaluated using real-life COVID-19 epidemiological data from the Public Health Agency of Canada (PHAC), accessed via Statistics Canada. As of November 12, 2020, the dataset included 209,811 confirmed cases across 11 key dimensions, including region, episode week, gender, age group, occupation, asymptomatic status, hospital status, transmission method, and clinical outcome.
Key Insights and Predictive Results
A major finding of the study is the power of generalization. By rolling up specific data points into broader categories, the machine learning algorithms became more effective, reducing "noise" and ensuring statistical significance. Furthermore, the treatment of NULL values proved critical; in the Canadian dataset, 19.4% of transmission methods were unstated. Including these values provided a more complete and honest picture of the epidemiological landscape.
The tool successfully generated several high-confidence predictive rules:
- Recovery Prediction: The rule
{community exposures, not hospitalized} → recoveredwas supported by 92,559 cases with 80% confidence. - Travel-Related Outcomes: The rule
{travel exp., not hospitalized} → recoveredshowed a 97.4% confidence level. - Demographic Trends: Analysis revealed that while gender did not significantly impact outcomes, age did. The "20s" age group had the highest absolute frequency of cases (19.12%), though older groups showed different recovery trajectories.
Conclusion and Future Work
While currently focused on textual epidemiological data, the framework demonstrates significant potential for big data analytics. Future work includes integrating visual analytics for easier exploration by non-technical users, applying the OLAP-ML framework to other domains like finance or transportation, and moving toward broader trend projection.
Summary updated