Optimizing the Union of Intersections LASSO (UoILASSO) and Vector Autoregressive (UoIVAR) Algorithms for Improved Statistical Estimation at Scale

Mahesh Balasubramanian, Trevor Ruiz, Brandon Cook, Sharmodeep Bhattacharyya, Prabhat, Aviral Shrivastava, Kristofer E. BouchardarXiv 2018 · OpenReview

Abstract

The analysis of scientific data of increasing size and complexity requires statistical machine learning methods that are both interpretable and predictive. Union of Intersections (UoI), a recently developed framework, is a two-step approach that separates model selection and model estimation. A linear regression algorithm based on UoI, $UoI_{LASSO}$, simultaneously achieves low false positives and low false negative feature selection as well as low bias and low variance estimates. Together, these qualities make the results both predictive and interpretable. In this paper, we optimize the $UoI_{LASSO}$ algorithm for single-node execution on NERSC's Cori Knights Landing, a Xeon Phi based supercomputer. We then scale $UoI_{LASSO}$ to execute on cores ranging from 68-278,528 cores on a range of dataset sizes demonstrating the weak and strong scaling of the implementation. We also implement a variant of $UoI_{LASSO}$, $UoI_{VAR}$ for vector autoregressive models, to analyze high dimensional time-series data. We perform single node optimization and multi-node scaling experiments for $UoI_{VAR}$ to demonstrate the effectiveness of the algorithm for weak and strong scaling. Our implementations enable the estimation of the largest VAR model (1000 nodes) we are aware of, and apply it to large neurophysiology data (192 nodes).

Introduction

In the era of "Big Science," datasets from fields like genomics and neuroscience are expanding into the terabyte and petabyte range. Researchers often face a trade-off between "black-box" machine learning models that offer high accuracy but low interpretability, and simpler statistical models that are easy to understand but often lack predictive power or fail at scale. The Union of Intersections (UoI) framework was designed to break this trade-off by separating model selection from model estimation. However, the original UoI algorithms were computationally expensive and not designed for massive parallelization. This paper details the optimization of these high-quality statistical tools for modern supercomputers to handle massive, high-dimensional scientific data.

The Union of Intersections (UoI) Framework

The UoI process involves two primary modules designed to ensure that the resulting model represents the underlying physical reality of the data rather than noise.

The UoI framework workflow, illustrating the two-step process of model selection via intersection and model estimation via union.
  1. Model Selection (Intersection): The algorithm runs numerous "bootstraps" (subsamples of data) across different regularization parameters. It only retains features that consistently appear across these samples (the intersection), which drastically reduces false positives.
  2. Model Estimation (Union): It then calculates nearly unbiased estimates (using Ordinary Least Squares) for these stable features and averages them (the union) to improve predictive accuracy and reduce variance.

Optimization and Scaling Strategies

To enable UoI to handle massive scientific datasets, the authors optimized two specific implementations: $UoI_{LASSO}$ (for sparse linear regression) and $UoI_{VAR}$ (for time-series analysis). Key technical optimizations included:

  • Linear Algebra: Native library calls were replaced with the Intel Math Kernel Library (MKL), utilizing AVX-512 instructions for a significant boost in single-node performance.
  • Parallelization: A multi-tier parallelization strategy using MPI was implemented, distributing work across bootstrap samples ($P_B$) and regularization parameters ($P_\lambda$).
  • Data I/O: A novel randomized data distribution technique for HDF5 files was developed, allowing thousands of cores to read and reshuffle data efficiently without encountering I/O bottlenecks.

Performance Results

The framework was tested on synthetic datasets ranging from 16GB to 8TB to evaluate both weak and strong scaling. $UoI_{LASSO}$ was successfully scaled to 278,528 cores on NERSC’s Cori supercomputer, demonstrating nearly ideal weak scaling.

Scaling performance of UoILASSO across various data sizes and parallelization strategies, highlighting the balance between computation and communication time.

As illustrated in Figure 3, the balance between computation and communication is critical at scale. At extreme core counts (10,000+), the time spent moving data between cores (MPI_Allreduce) becomes the primary limiting factor as the actual computation becomes increasingly fast.

Real-World Application in Neuroscience

The team applied the optimized $UoI_{VAR}$ to a 1.3TB neurophysiology dataset consisting of neural spikes recorded from 192 electrodes in a non-human primate. Executing on 81,600 cores, the algorithm provided a stable map of neural interactions that would be computationally infeasible with standard methods. Furthermore, the team estimated a 1,000-node VAR model, which is believed to be the largest of its kind ever reported.

Conclusion

This work demonstrates that the UoI framework can be effectively scaled to meet the demands of modern scientific data analysis. By optimizing for HPC architectures, the researchers have enabled the use of interpretable and predictive models at the terabyte scale. Future work will focus on communication-avoiding algorithms to mitigate bottlenecks at extreme core counts and addressing the cubic memory growth required for high-dimensional VAR models.

Summary updated

Abstract

The analysis of scientific data of increasing size and complexity requires statistical machine learning methods that are both interpretable and predictive. Union of Intersections (UoI), a recently developed framework, is a two-step approach that separates model selection and model estimation. A linear regression algorithm based on UoI, $UoI_{LASSO}$, simultaneously achieves low false positives and low false negative feature selection as well as low bias and low variance estimates. Together, these qualities make the results both predictive and interpretable. In this paper, we optimize the $UoI_{LASSO}$ algorithm for single-node execution on NERSC's Cori Knights Landing, a Xeon Phi based supercomputer. We then scale $UoI_{LASSO}$ to execute on cores ranging from 68-278,528 cores on a range of dataset sizes demonstrating the weak and strong scaling of the implementation. We also implement a variant of $UoI_{LASSO}$, $UoI_{VAR}$ for vector autoregressive models, to analyze high dimensional time-series data. We perform single node optimization and multi-node scaling experiments for $UoI_{VAR}$ to demonstrate the effectiveness of the algorithm for weak and strong scaling. Our implementations enable to use estimate the largest VAR model (1000 nodes) we are aware of, and apply it to large neurophysiology data 192 nodes).
BibTeX
@misc{balasubramanian2018optimizing,
  author = {Mahesh Balasubramanian and Trevor Ruiz and Brandon Cook and Sharmodeep Bhattacharyya and Prabhat and Aviral Shrivastava and Kristofer E. Bouchard},
  title = {{Optimizing the Union of Intersections LASSO (UoILASSO) and Vector Autoregressive (UoIVAR) Algorithms for Improved Statistical Estimation at Scale}},
  year = {2018},
  eprint = {1808.06992},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.06992},
}
Figure