On the intrinsic dimensionality of Covid-19 data: a global perspective

Abhishek Varghese, Edgar Santos–Fernández, Francesco Denti, Antonietta Mira, Kerrie MengersenarXiv 2022 · DOI · arXiv

Abstract

This paper aims to develop a global perspective of the complexity of the relationship between the standardised per-capita growth rate of Covid-19 cases, deaths, and the OxCGRT Covid-19 Stringency Index, a measure describing a country's stringency of lockdown policies. To achieve our goal, we use a heterogeneous intrinsic dimension estimator implemented as a Bayesian mixture model, called Hidalgo. We identify that the Covid-19 dataset may project onto two low-dimensional manifolds without significant information loss. The low dimensionality suggests strong dependency among the standardised growth rates of cases and deaths per capita and the OxCGRT Covid-19 Stringency Index for a country over 2020-2021. Given the low dimensional structure, it may be feasible to model observable Covid-19 dynamics with few parameters. Importantly, we identify spatial autocorrelation in the intrinsic dimension distribution worldwide. Moreover, we highlight that high-income countries are more likely to lie on low-dimensional manifolds, likely arising from aging populations, comorbidities, and increased per capita mortality burden from Covid-19. Finally, we temporally stratify the dataset to examine the intrinsic dimension at a more granular level throughout the Covid-19 pandemic.

Introduction

Modeling a global pandemic presents a significant challenge due to the high-dimensional nature of the data. Between daily case counts, mortality rates, and varying government lockdown policies across hundreds of countries, the "nominal dimension"—the total number of variables recorded—is massive. This study addresses the challenge of Intrinsic Dimensionality (ID): identifying the minimum number of independent variables required to represent a dataset without significant information loss.

While traditional statistical methods often assume a single ID for an entire dataset, global pandemic data is inherently heterogeneous. This research seeks to determine if different countries follow distinct "complexity rules" and whether these rules correlate with geography, income levels, or demographic factors.

Methodology: The Hidalgo Algorithm

The core methodology involves the application of Hidalgo (Heterogeneous Intrinsic Dimension Algorithm). Unlike standard dimensionality reduction techniques like PCA, which seek global patterns, Hidalgo is a Bayesian mixture model capable of segmenting a dataset into multiple groups, each characterized by its own local intrinsic dimension.

Building on the "Two Nearest Neighbor" (TWO-NN) estimator, Hidalgo analyzes the ratio of distances between a data point and its two closest neighbors to estimate the dimension of the manifold on which the data resides. The model utilizes Markov Chain Monte Carlo (MCMC) techniques to cluster countries sharing similar data complexity, revealing underlying shared pandemic dynamics.

Data Description and Temporal Stratification

The study utilized three primary time-series datasets spanning March 1, 2020, to May 29, 2021: Covid-19 new cases (standardized per million), Covid-19 deaths (standardized per million), and the OxCGRT Stringency Index (CSI), which quantifies lockdown strictness on a scale of 0–100.

Sample of the standardized Covid-19 dataset structure used for analysis

After filtering for data availability and population size (>1 million), the final analysis included 115 countries. The nominal dimension of this combined dataset was $D = 1362$, representing 454 days of measurements across the three variables. To examine how complexity evolved, the researchers stratified the data into four distinct temporal stages.

Temporal stratification stages for the Covid-19 pandemic analysis

Key Results and Discussion

Massive Redundancy and Manifold Discovery

Despite the high nominal dimensionality, the pandemic's "true" complexity is remarkably low. The global data projects onto two manifolds with posterior mean dimensions of $d_1 = 12$ and $d_2 = 9$. This represents a dimensionality reduction of over 99%, suggesting that the relationship between cases, deaths, and policy is highly predictable and can be modeled with few parameters.

The Demographic and Income Divide

A significant finding is the link between national income and data complexity. High-income countries are more likely to lie on the lower-dimensional manifold ($d=9$). This is attributed to older populations and higher rates of comorbidities; in these settings, mortality is more tightly coupled to case rates, reducing the "degrees of freedom" in the data. Specifically, countries in the high-ID manifold ($d=12$) had an average of less than 7% of their population aged 65+, whereas low-ID manifold countries averaged 13%.

Spatial and Temporal Dynamics

Even without explicit geographical input, the model identified spatial autocorrelation in pandemic complexity. A Moran’s I test yielded a high value of 0.85, proving that complexity spreads geographically, likely due to human mobility and shared demographic factors. Temporally, Stage 1 (early 2020) showed a more unified global response, while later stages saw countries diverge into multiple distinct manifolds as local policies and variants emerged.

Conclusion and Limitations

The low intrinsic dimensionality of Covid-19 data suggests that practitioners can model observable dynamics using very few parameters, which is vital for real-time forecasting. However, the authors acknowledge limitations, including potential data quality issues (under-reporting in low-income settings) and the model's assumption of independent observations, despite the inherent temporal autocorrelation of pandemic data. Future work should incorporate more granular drivers, such as vaccination rates and specific healthcare resource levels, to further refine these complexity models.

Summary updated

Abstract

This paper aims to develop a global perspective of the complexity of the relationship between the standardised per-capita growth rate of Covid-19 cases, deaths, and the OxCGRT Covid-19 Stringency Index, a measure describing a country's stringency of lockdown policies. To achieve our goal, we use a heterogeneous intrinsic dimension estimator implemented as a Bayesian mixture model, called Hidalgo. We identify that the Covid-19 dataset may project onto two low-dimensional manifolds without significant information loss. The low dimensionality suggests strong dependency among the standardised growth rates of cases and deaths per capita and the OxCGRT Covid-19 Stringency Index for a country over 2020-2021. Given the low dimensional structure, it may be feasible to model observable Covid-19 dynamics with few parameters. Importantly, we identify spatial autocorrelation in the intrinsic dimension distribution worldwide. Moreover, we highlight that high-income countries are more likely to lie on low-dimensional manifolds, likely arising from aging populations, comorbidities, and increased per capita mortality burden from Covid-19. Finally, we temporally stratify the dataset to examine the intrinsic dimension at a more granular level throughout the Covid-19 pandemic.
BibTeX
@misc{varghese2022intrinsic,
  author = {Abhishek Varghese and Edgar Santos–Fernández and Francesco Denti and Antonietta Mira and Kerrie Mengersen},
  title = {{On the intrinsic dimensionality of Covid-19 data: a global perspective}},
  year = {2022},
  eprint = {2203.04165},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.04165},
}
Figure