From 5Vs to 6Cs: Operationalizing Epidemic Data Management with COVID-19 Surveillance

Akhil Sai Peddireddy, Dawen Xie, Pramod Patil, Mandy Wilson, Dustin Machi, Srinivasan Venkatramanan, Brian Klahn, Przemyslaw Porebski, Parantapa Bhattacharya, Shirish Dumbre, Erin Raymond, Madhav MarathemedRxiv 2020 · DOI

Abstract

The COVID-19 pandemic brought to the forefront an unprecedented need for experts, as well as citizens, to visualize spatio-temporal disease surveillance data. Web application dashboards were quickly developed to fill this gap, including those built by JHU, WHO, and CDC, but all of these dashboards supported a particular niche view of the pandemic (ie, current status or specific regions). In this paper, we describe our work developing our own COVID-19 Surveillance Dashboard, which offers a universal view of the pandemic while also allowing users to focus on the details that interest them. From the beginning, our goal was to provide a simple visual way to compare, organize, and track near-real-time surveillance data as the pandemic progresses. Our dashboard includes a number of advanced features for zooming, filtering, categorizing and visualizing multiple time series on a single canvas. In developing this dashboard, we have also identified 6 key metrics we call the 6Cs standard which we propose as a standard for the design and evaluation of real-time epidemic science dashboards.

Introduction: The Need for a Gold Standard

During the early stages of the COVID-19 pandemic, data was ubiquitous but highly fragmented. While organizations like Johns Hopkins University (JHU), the WHO, and the CDC launched vital dashboards, they often focused on specific niches or lacked deep historical and sub-national data. This fragmentation highlighted a critical gap: the lack of a "gold standard" for epidemic data management. Researchers and policymakers require data that is not merely a snapshot, but a reliable, time-series-ready resource. The challenge lies in the inherent "messiness" of global reporting, characterized by inconsistent naming conventions, missing recovery statistics, and frequent changes in data sharing formats (e.g., transitioning from structured CSVs to unstructured PDF reports).

The 6Cs Framework

To address these challenges, we propose the 6Cs standard—Consistent, Correct, Current, Comprehensive, Curated, and Computer-readable—as a framework for real-time epidemic science. This framework evolves traditional "Big Data" concepts (the 5Vs: Volume, Velocity, Variety, Veracity, and Value) into operational requirements for public health surveillance.

The 6Cs framework maps traditional Big Data '5Vs' to operational epidemic data requirements.

The 6Cs are defined as follows:

  1. Consistent: Uniform data formats and stable historical records.
  2. Correct: Rigorous validation to ensure sub-regional data sums to totals and counts do not erroneously decrease.
  3. Current: Automated, high-frequency updates to reflect the latest situation.
  4. Comprehensive: Inclusion of derived metrics like "active cases" and population-normalized "per 100k" numbers.
  5. Curated: Sourcing data at multiple spatial resolutions (country, state, county).
  6. Computer-readable: Providing data via APIs or structured files (CSV) to facilitate downstream modeling.

System Architecture and Implementation

To operationalize the 6Cs, we developed a three-layered data pipeline designed to handle the complexities of global pandemic data.

The three-layered data pipeline: Curation, Core, and Augmented layers.
  • Curation Layer: Automated scrapers pull data hourly from diverse sources, including 1Point3Acres, USAFacts, the WHO, and various national health ministries.
  • Core Layer: This layer standardizes data into a spatial hierarchy (Global -> Admin1 -> County) and maps regions to standardized ISO/FIPS codes. It includes manual "human-in-the-loop" validation to catch anomalies, such as large retrospective data revisions.
  • Augmented Layer: This layer calculates derived metrics. A key innovation is our recovery estimation algorithm. Since many regions do not report recovery data, we developed a formula based on clinical studies to estimate active cases by weighting confirmed cases from 14, 28, and 42 days prior.

Impact and Utility

The UVA COVID-19 Surveillance Dashboard has seen massive adoption. As of late 2020, it had served over 1.13 million users from 220 countries and processed over 60 million requests. Beyond public use, the platform supports federal agencies (DoD, CDC) and state authorities (Virginia Department of Health) in predictive modeling and resource allocation.

The system also features advanced analytics, including a "4W1H" (Which, What, Where, When, How) natural language query interface. This allows users to ask complex questions like "Top 5 incidence deaths in Virginia on April 15" and receive immediate visual answers, bridging the gap between raw data and actionable insights.

Conclusion and Future Work

While the dashboard provides extensive global coverage, expanding spatial resolution remains a challenge. Sub-national data is currently limited to approximately 20 countries due to the difficulty of sourcing and mapping inconsistent local data. Future efforts will focus on expanding the natural language query system to handle complex temporal questions and incorporating more diverse data types, such as test positivity rates and 7-day moving averages, into the automated analytics engine.

The dashboard is available at: https://nssac.bii.virginia.edu/covid-19/dashboard/

Summary updated

Abstract

The COVID-19 pandemic brought to the forefront an unprecedented need for experts, as well as citizens, to visualize spatio-temporal disease surveillance data. Web application <i>dashboards</i> were quickly developed to fill this gap, including those built by JHU, WHO, and CDC, but all of these dashboards supported a particular niche view of the pandemic (ie, current status or specific regions). In this paper, we describe our work developing our own COVID-19 Surveillance Dashboard, available at https://nssac.bii.virginia.edu/covid-19/dashboard/, which offers a universal view of the pandemic while also allowing users to focus on the details that interest them. From the beginning, our goal was to provide a simple visual way to compare, organize, and track near-real-time surveillance data as the pandemic progresses. Our dashboard includes a number of advanced features for zooming, filtering, categorizing and visualizing multiple time series on a single canvas. In developing this dashboard, we have also identified 6 key metrics we call <i>the 6Cs standard</i> which we propose as a standard for the design and evaluation of real-time epidemic science dashboards. Our dashboard was one of the first released to the public, and remains one of the most visited and highly used. Our group uses it to support federal, state and local public health authorities, and it is used by people worldwide to track the pandemic evolution, build their own dashboards, and support their organizations as they plan their responses to the pandemic. We illustrate the utility of our dashboard by describing how it can be used to support data story-telling - an important emerging area in data science.
BibTeX
@misc{peddireddy2020from,
  author = {Akhil Sai Peddireddy and Dawen Xie and Pramod Patil and Mandy Wilson and Dustin Machi and Srinivasan Venkatramanan and Brian Klahn and Przemyslaw Porebski and Parantapa Bhattacharya and Shirish Dumbre and Erin Raymond and Madhav Marathe},
  title = {{From 5Vs to 6Cs: Operationalizing Epidemic Data Management with COVID-19 Surveillance}},
  year = {2020},
  doi = {10.1101/2020.10.27.20220830},
  howpublished = {medRxiv},
}
Figure