SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

Beidi Chen, Tharun Medini, Anshumali ShrivastavaMLSys 2019 · OpenReview

TL;DR

This paper introduces SLIDE, a novel deep learning engine that achieves significant speedups over hardware-accelerated frameworks like TensorFlow on GPUs by employing smart, randomized algorithms and efficient CPU parallelism. SLIDE demonstrates that algorithmic innovation can outperform brute-force hardware acceleration, achieving up to 3.5x faster training than TF-GPU and 10x faster than TF-CPU on large-scale datasets.

What Problem Does This Paper Address?

Training large-scale deep learning models, with their hundreds of millions of parameters, demands immense computational resources. The community has largely relied on specialized hardware, particularly GPUs, to accelerate this process. However, this hardware is expensive, difficult to generalize, and algorithmic advancements have struggled to keep pace. The authors identify a gap where algorithmic approaches haven't demonstrated a clear advantage over powerful hardware, and propose SLIDE to challenge this status quo by optimizing the algorithms themselves for efficient CPU execution.

Methods

SLIDE's core innovation lies in its blend of smart randomized algorithms and multi-core CPU parallelism.

  • Locality Sensitive Hashing (LSH) for Adaptive Sparsity: Instead of computing all neuron activations, SLIDE uses LSH to efficiently identify and activate only a sparse subset of relevant neurons in each layer. This drastically reduces computation.
  • Asynchronous SGD: The sparse gradient updates enabled by LSH allow for asynchronous Stochastic Gradient Descent (SGD), minimizing synchronization overhead and maximizing CPU utilization.
  • Optimized Sampling Strategies: The paper explores and implements three strategies for selecting active neurons: Vanilla Sampling, TopK Sampling, and Hard Thresholding, with Vanilla Sampling being the most efficient.
  • CPU Optimizations: SLIDE leverages OpenMP for parallelization, and incorporates techniques like Transparent Hugepages and SIMD instructions (Intel AVX) to reduce cache misses and improve memory access patterns.

Datasets

The paper evaluates SLIDE on two large-scale, real-world datasets:

  • Delicious-200K: A dataset from the Extreme Classification Repository, derived from social bookmarking systems.
  • Amazon-670K: A product-to-product recommendation dataset.

These datasets are characterized by their large number of features and labels, making them suitable for testing extreme classification scenarios.

Key Insights and Ideas

  • Algorithmic Superiority: The most striking insight is that a well-designed algorithm running on a standard CPU can outperform highly optimized GPU implementations for certain tasks, challenging the assumption that hardware acceleration is always the bottleneck.
  • Sparsity is Key: By adaptively selecting only a small fraction of active neurons, SLIDE dramatically reduces computational load, especially in the final layers of large networks where most computation occurs.
  • Asynchronous Parallelism Unleashed: The sparse nature of SLIDE's computations enables highly efficient asynchronous parallel processing on CPUs, leading to near-perfect scaling with increasing core counts.
  • CPU Bottleneck Shift: While TensorFlow on CPU becomes increasingly memory-bound with more cores, SLIDE's memory bottleneck reduces, allowing it to better leverage multi-core architectures.

Key Results

  • Speedup: SLIDE achieves up to 3.5x faster training than TensorFlow on a V100 GPU and over 10x faster than TensorFlow on a CPU for comparable accuracy on large-scale datasets.
  • Efficiency: On the Delicious-200K dataset, SLIDE is 2.7x faster than TF-GPU. On the larger Amazon-670K dataset, it is 1.8x faster than TF-GPU.
  • Scalability: SLIDE exhibits excellent scalability on CPUs, outperforming TF-CPU with as few as 8 cores on Delicious-200K and 2 cores on Amazon-670K.
  • Convergence: SLIDE's adaptive sampling does not negatively impact convergence, achieving similar iteration-wise accuracy to baselines.

Limitations and Future Work

  • Current Focus: SLIDE is currently optimized for fully connected networks.
  • Future Work: The authors propose extending SLIDE to include convolutional layers, which would be a significant step towards broader applicability. They also anticipate that a distributed implementation of SLIDE would be highly appealing due to its minimal communication costs stemming from sparse gradients.

Summary updated

Abstract

Deep Learning (DL) algorithms are the central focus of modern machine learning systems. As data volumes keep growing, it has become customary to train large neural networks with hundreds of millions of parameters to maintain enough capacity to memorize these volumes and obtain state-of-the-art accuracy. To get around the costly computations associated with large models and data, the community is increasingly investing in specialized hardware for model training. However, specialized hardware is expensive and hard to generalize to a multitude of tasks. The progress on the algorithmic front has failed to demonstrate a direct advantage over powerful hardware such as NVIDIA-V100 GPUs. This paper provides an exception. We propose SLIDE (Sub-LInear Deep learning Engine) that uniquely blends smart randomized algorithms, with multi-core parallelism and workload optimization. Using just a CPU, SLIDE drastically reduces the computations during both training and inference outperforming an optimized implementation of Tensorflow (TF) on the best available GPU. Our evaluations on industry-scale recommendation datasets, with large fully connected architectures, show that training with SLIDE on a 44 core CPU is more than 3.5 times (1 hour vs. 3.5 hours) faster than the same network trained using TF on Tesla V100 at any given accuracy level. On the same CPU hardware, SLIDE is over 10x faster than TF. We provide codes and scripts for reproducibility.
BibTeX
@inproceedings{chen2019slide,
  author = {Beidi Chen and Tharun Medini and Anshumali Shrivastava},
  title = {{SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems}},
  booktitle = {MLSys},
  year = {2019},
  url = {https://openreview.net/forum?id=GG_QNMNZbQ7},
}
Figure