SMALE: Enhancing Scalability of Machine Learning Algorithms on Extreme-Scale Computing Platforms

Yiran Chen, Hai Li2022 · DOI

Abstract

Deployment and execution of machine learning tasks on extreme-scale computing platforms face several significant technical challenges: 1) High computing cost incurred by dense networks – The computing workload of deep networks with densely-connected topology increases rapidly with the network size, imposing a non-scalable computing model of extreme-scale computing platforms; 2) Non-optimized workload distribution – Many advanced deep learning algorithms, e.g., sparsification and irregular net-work topology, produce very unbalanced workload distribution on extreme-scale computing platforms. The computation efficiency is greatly hindered by the incurred data and computation redundancies as well as long tails of the node with extensive workload; 3) Constraints in data movement and I/O bottle-neck – Inter-node data movement in extreme-scale computing platforms are associated with high energy and latency costs, and subject to the constraints of I/O bandwidth; and 4) Generalization of algorithm realization and acceleration on computing platforms – The large varieties of machine learning algorithms and structures of extreme-scale computing platforms make the derivation of a generalized algorithm realization and acceleration method very challenging, which, however, is the requirement by domain scientists and interested users. We call the above challenges Smale’s Problems in Machine Learning and Understanding for High-Performance Computing Scientific Discovery. The objective of our three-year research project is to develop a holistic innovation set at structure, assembly, and acceleration layers of machine learning algorithms to address the above challenges in algorithm deployment and execution. Three tasks are particularly performed, including: At the algorithm structure level, we investigate the techniques that can structurally sparsify on the topology of deep networks for computing workload reduction. We also study clustering and pruning techniques that can optimize the workload distributions over the extreme-scale computing platforms; At the algorithm assembly level, we derive a unified learning framework for unsupervised transfer learning and dynamic growing capabilities. Novel training methods are also exploited to enhance the training efficiency of the proposed framework; At the algorithm acceleration level, we will develop a series of techniques that can accelerate the computation of sparse matrix operations, which are one of the core executions in deep learning and optimize memory access of the concerned platforms. Our proposed techniques attack the fundamental problems in machine learning algorithms running on extreme-scale computing platforms by vertically integrating the solutions at three closely entangled layers, paving the long-term scaling path of machine learning applications under DOE context.

1. Introduction: Addressing the "Scalability Wall"

As deep neural networks (DNNs) grow in complexity, they face diminishing returns on extreme-scale hardware. The SMALE project identifies four "Smale’s Problems" in High-Performance Computing (HPC) that hinder scalability: high density costs, unbalanced workloads, the "memory wall" of I/O bottlenecks, and a generalization gap across heterogeneous hardware. To overcome these, the project proposes a three-pronged vertical integration strategy across algorithm structure, assembly, and hardware acceleration.

2. Methodology

2.1 Algorithm Structure: SVD Training

Instead of the traditional approach of training dense networks and pruning them post-hoc, SMALE investigates structural sparsification from the outset. By decomposing weight matrices into Singular Value Decomposition (SVD) form, the framework trains low-rank components directly. This method utilizes "Singular Vector Orthogonality Regularization" to maintain gradient stability, achieving the benefits of low-rank models without the computational overhead of constant decomposition during the training process.

SVD Training and Compression Workflow

2.2 Algorithm Assembly: AutoGrow

To eliminate the manual trial-and-error associated with finding optimal network depth, the researchers developed AutoGrow. This system starts with a shallow "seed" backbone and automatically grows sub-modules (such as residual blocks) until a stopping policy is met. This automated discovery process ensures the architecture is optimized for the specific complexity of the target dataset.

AutoGrow: Automated Network Depth Discovery

2.3 Algorithm Acceleration

At the acceleration layer, the project introduces hardware-specific optimizations to mitigate data movement costs:

  • STT-RAM in GPUs: Replacing power-hungry SRAM with Spin-Transfer Torque RAM for register files to reduce leakage power.
  • NeuralHMC: An accelerator based on Hybrid Memory Cube (HMC) technology that utilizes "Process-In-Memory" (PIM) to minimize data movement.
  • GraphR: A ReRAM-based accelerator specifically for graph processing, converting vertex programs into efficient sparse matrix-vector multiplications.
  • HyPar: A dynamic programming approach to determine the optimal mix of data and model parallelism for each network layer.

3. Key Insights and Results

  • Efficiency Gains: SVD training achieved a 6.42x speedup on ResNet-110 (CIFAR-10) with minimal accuracy loss compared to training from scratch.
  • Hardware Energy Reduction: Using STT-RAM for shared memory led to a 49.3% energy reduction and a 116% increase in energy efficiency.
  • Specialized Speedups: The ReGAN accelerator (for GANs) achieved a 240x performance speedup over traditional GPU platforms, while GraphR achieved a 16.01x average speedup over CPU baselines.
  • The Memory-Compute Mismatch: Profiling revealed that while GPU memory is often 90% utilized, computing units are only ~30% utilized as they wait for data, justifying the focus on PIM and emerging memory technologies.

4. Conclusion

The SMALE project demonstrates that vertical integration—from low-rank structural training to automated architecture discovery and hardware-aware acceleration—is essential for scaling machine learning on extreme-scale platforms. Future work will focus on refining these techniques in collaboration with Oak Ridge National Laboratory (ORNL) to support Department of Energy (DOE) scientific discovery tasks.

Summary updated

Abstract

Deployment and execution of machine learning tasks on extreme-scale computing platforms face several significant technical challenges: 1) High computing cost incurred by dense networks – The computing workload of deep networks with densely-connected topology increases rapidly with the network size, imposing a non-scalable computing model of extreme-scale computing platforms; 2) Non-optimized workload distribution – Many advanced deep learning algorithms, e.g., sparsification and irregular net-work topology, produce very unbalanced workload distribution on extreme-scale computing platforms. The computation efficiency is greatly hindered by the incurred data and computation redundancies as well as long tails of the node with extensive workload; 3) Constraints in data movement and I/O bottle-neck – Inter-node data movement in extreme-scale computing platforms are associated with high energy and latency costs, and subject to the constraints of I/O bandwidth; and 4) Generalization of algorithm realization and acceleration on computing platforms – The large varieties of machine learning algorithms and structures of extreme-scale computing platforms make the derivation of a generalized algorithm realization and acceleration method very challenging, which, however, is the requirement by domain scientists and interested users. We call the above challenges Smale’s Problems in Machine Learning and Understanding for High-Performance Computing Scientific Discovery. The objective of our three-year research project is to develop a holistic innovation set at structure, assembly, and acceleration layers of machine learning algorithms to address the above challenges in algorithm deployment and execution. Three tasks are particularly performed, including: At the algorithm structure level, we investigate the techniques that can structurally sparsify on the topology of deep networks for computing workload reduction. We also study clustering and pruning techniques that can optimize the workload distributions over the extreme-scale computing platforms; At the algorithm assembly level, we derive a unified learning framework for unsupervised transfer learning and dynamic growing capabilities. Novel training methods are also exploited to enhance the training efficiency of the proposed framework; At the algorithm acceleration level, we will develop a series of techniques that can accelerate the computation of sparse matrix operations, which are one of the core executions in deep learning and optimize memory access of the concerned platforms. Our proposed techniques attack the fundamental problems in machine learning algorithms running on extreme-scale computing platforms by vertically integrating the solutions at three closely entangled layers, paving the long-term scaling path of machine learning applications under DOE context. Three tasks corresponding to the above respective research orientations are performed during the three-year project period with our collaborators at ORNL. The outcome of the proposed project is anticipated to form a holistic solution set of novel algorithms and network topologies, efficient training techniques, and fast acceleration methods to promote the computing scalability of the machine learning applications of particular interest to DOE.
BibTeX
@misc{chen2022smale,
  author = {Yiran Chen and Hai Li},
  title = {{SMALE: Enhancing Scalability of Machine Learning Algorithms on Extreme-Scale Computing Platforms}},
  year = {2022},
  doi = {10.2172/1846568},
}
Figure