High Performance Multivariate Geospatial Statistics on Manycore Systems

Mary Lai O. Salvaña, Sameh Abdulah, Huang Huang, Hatem Ltaief, Ying Sun, Marc G. Genton, David E. KeyesIEEE TPDS 2021 · DOI

TL;DR

This paper tackles the computational bottleneck in multivariate geospatial statistics, specifically the expensive Gaussian log-likelihood function required for techniques like cokriging. They introduce a high-performance framework leveraging Tile Low-Rank (TLR) approximations and a task-based programming model to significantly speed up these calculations on manycore systems, while also proposing a novel method to assess the accuracy of these approximations.

What Problem Does This Paper Address?

Geospatial statistics plays a crucial role in understanding and predicting environmental phenomena by modeling spatial relationships in data. Multivariate geospatial models, which consider multiple correlated variables simultaneously (e.g., temperature, humidity, wind speed at various locations), offer more accurate predictions than univariate models. However, these models rely on computing the Gaussian log-likelihood function, which involves computationally intensive matrix operations (specifically, inverting and calculating the determinant of a large covariance matrix). This computational burden has historically limited their application to large-scale datasets, which are increasingly common due to advancements in data collection technologies. The paper aims to bridge this gap by enabling efficient and accurate multivariate geospatial modeling on modern high-performance computing (HPC) architectures.

Methods

The core of the paper's contribution lies in its approach to tackling the computational complexity of the Gaussian log-likelihood function. They employ two main strategies:

  • Tile Low-Rank (TLR) Approximation: Instead of computing the full covariance matrix, TLR approximates the off-diagonal blocks of the matrix with low-rank representations. This significantly reduces the memory footprint and computational cost. The paper utilizes the CHAMELEON library for dense linear algebra and HICMA for TLR computations, both managed by the STARPU dynamic runtime system.
  • Task-Based Programming Model: The STARPU runtime system is used to manage asynchronous computational tasks, allowing for efficient execution on heterogeneous HPC architectures (multi-core CPUs, GPUs). This model breaks down the problem into a Directed Acyclic Graph (DAG) of tasks, enabling dynamic scheduling and efficient resource utilization.

Additionally, the paper introduces a novel multivariate assessment algorithm for evaluating the accuracy of the TLR approximation. This algorithm extends existing univariate metrics (MLOE/MMOM) to the multivariate setting, using cross-covariance functions to assess prediction accuracy.

Datasets

The paper utilizes both synthetic and real-world datasets for evaluation:

  • Synthetic Datasets: Generated using a parsimonious bivariate Matérn cross-covariance function to simulate multivariate Gaussian random fields. These datasets allow for controlled experiments to assess parameter estimation accuracy and prediction performance under varying spatial dependence strengths.
  • Real Datasets:
    • A bivariate dataset of zonal (U) and meridional (V) wind components from a Weather Research and Forecasting (WRF) model simulation over the Arabian Sea.
  • A trivariate dataset including U and V wind components and temperature, also from the WRF model over the Arabian Sea.

The paper does not introduce new benchmark datasets but rather applies its framework to existing types of environmental data.

Key Insights and Ideas

  • TLR Approximation is Crucial for Scalability: The paper strongly demonstrates that TLR approximations are essential for handling large-scale multivariate geospatial problems, offering substantial speedups and memory savings compared to exact computations.
  • Accuracy vs. Performance Trade-off: While TLR approximations offer significant performance gains, there's a trade-off with accuracy. The paper shows that higher accuracy levels (e.g., TLR9) are needed for stronger spatial dependencies to remain competitive with exact methods.
  • Novel Multivariate Accuracy Assessment: The proposed multivariate MLOE/MMOM criteria provide a more robust way to evaluate the performance of approximated models in a multivariate context, going beyond simple averaging of univariate metrics.
  • Task-Based Runtime is Key for Heterogeneity: The use of STARPU highlights the importance of dynamic runtime systems for effectively utilizing diverse HPC architectures (CPUs, GPUs) and managing complex task dependencies.
  • Cokriging Benefits from High Colocated Dependence: The experiments with synthetic data reveal that the gains from multivariate modeling (cokriging) over univariate modeling (kriging) are most pronounced when variables are highly correlated at the same location.

Key Results

  • Performance Speedup: The TLR-based approximation achieved speedups of up to 10X and 2X over exact MLE computations on various shared-memory architectures. For instance, on an AMD EPYC system, an average speedup of 6X was observed.
  • Memory Reduction: TLR representations required significantly less memory than dense representations, with savings of 6.68X, 4.93X, and 3.86X for TLR5, TLR7, and TLR9, respectively, for large problem sizes.
  • Accuracy Preservation: The proposed multivariate assessment criteria (MLOE/MMOM) demonstrated that TLR approximations, especially at higher accuracy levels (TLR7 and TLR9), can preserve accuracy comparable to exact methods, particularly for strong spatial dependencies.
  • Parameter Estimation: The paper showed that TLR approximations can accurately estimate parameters for the parsimonious Matérn cross-covariance function, outperforming other approximation techniques like DST, especially when spatial dependence is strong.
  • Real Data Application: The framework was successfully applied to real environmental datasets (wind components and temperature), demonstrating its practical utility.

Limitations and Future Work

  • TLR Accuracy Dependence: The paper acknowledges that the effectiveness of TLR depends on the chosen accuracy level, and for strong spatial dependencies, higher accuracy (TLR9) is necessary to match exact methods.
  • Isotropy Assumption: For real data applications, the paper relies on the assumption of spatial isotropy (covariance depends only on distance, not location). While this was justified for the Arabian Sea dataset, it might not hold for all environmental phenomena.
  • Future Work: The authors suggest extending their framework to model and predict environmental variables that are indexed in both space and time, which would introduce further challenges related to problem dimensionality and prediction accuracy. They also mention exploring more advanced matrix compression strategies that might offer better arithmetic complexity.

Summary updated

BibTeX
@article{salvana2021high,
  author = {Mary Lai O. Salvaña and Sameh Abdulah and Huang Huang and Hatem Ltaief and Ying Sun and Marc G. Genton and David E. Keyes},
  title = {{High Performance Multivariate Geospatial Statistics on Manycore Systems}},
  journal = {IEEE Trans. Parallel Distributed Syst.},
  year = {2021},
  doi = {10.1109/tpds.2021.3071423},
}
Figure