Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework

Junxiang Wang, Zheng Chai, Yue Cheng, Liang Zhao

Abstract: Alternating Direction Method of Multipliers (ADMM) has recently been proposed as a potential alternative optimizer to the Stochastic Gradient Descent (SGD) for deep learning problems. This is because ADMM can solve gradient vanishing and poor conditioning problems. Moreover, it has shown good scalability in many large-scale deep learning applications. However, there still lacks a parallel ADMM computational framework for deep neural networks because of layer dependency among variables. In this paper, we propose a novel parallel deep learning ADMM framework (pdADMM) to achieve layer parallelism: parameters in each layer of neural networks can be updated independently in parallel. The convergence of the proposed pdADMM to a critical point is theoretically proven under mild conditions. The convergence rate of the pdADMM is proven to be $o(1/k)$ where $k$ is the number of iterations. Extensive experiments on six benchmark datasets demonstrated that our proposed pdADMM can lead to more than 10 times speedup for training large-scale deep neural networks, and outperformed most of the comparison methods.

Introduction: Breaking the Sequential Bottleneck

Standard deep learning training relies heavily on Stochastic Gradient Descent (SGD) and the backpropagation algorithm. While these methods have driven the current AI revolution, they suffer from a fundamental structural flaw: inherent sequentiality. To calculate gradients for any given layer, the system must wait for the results from the subsequent layer. This "locking" mechanism prevents true model parallelism, where different computing units could ideally process different layers of a deep network simultaneously.

Furthermore, gradient-based methods are frequently plagued by the "vanishing gradient" problem in very deep architectures and "poor conditioning," where minor input variations lead to unstable gradient swings. While the Alternating Direction Method of Multipliers (ADMM) has been explored as a gradient-free alternative to mitigate these issues, previous implementations still required layers to be updated one after another. This research introduces pdADMM, a framework designed to finally decouple these layer dependencies.

The pdADMM Framework

The core innovation of pdADMM lies in a mathematical reformulation of the feed-forward neural network optimization problem. By treating the connections between layers as constraints rather than direct functional dependencies, the authors enable simultaneous updates across the entire model architecture.

Figure 1: Conceptual overview of DNN reformulation into a parallel framework where layers can be updated independently.

Key Methodological Steps:

  1. Variable Splitting: The connection between layers is split into two independent variables ($q_l$ for output and $p_{l+1}$ for input). The physical connection is then treated as a mathematical constraint ($p_{l+1} = q_l$).
  2. Augmented Lagrangian: Using this method, the global training objective is decomposed into smaller, independent subproblems—one for each layer.
  3. Layer Parallelism: Because these subproblems are decoupled, different workers (such as separate GPUs or CPU cores) can solve them at the same time. A worker can update Layer 1 while another simultaneously updates Layer 10.
  4. Quadratic Approximation: To maintain efficiency and avoid expensive matrix inversions, the framework employs a quadratic approximation technique to simplify weight and input updates.

Experimental Evaluation and Results

The framework was rigorously tested on six image classification benchmarks: MNIST, Fashion MNIST, kMNIST, SVHN, CIFAR-10, and CIFAR-100. The results highlight a significant leap in training efficiency and scalability.

Figure 3: Speedup achieved by pdADMM compared to serial methods as the number of layers increases.

Performance Insights:

  • Linear Speedup Scaling: As shown in the experimental data, the speedup is not static; it scales linearly with the depth of the network. In tests with 19 hidden layers, the system achieved over 11x speedup, proving that the deeper the network, the greater the benefit of the pdADMM parallelization.
  • Immunity to Vanishing Gradients: Since pdADMM is gradient-free and solves local subproblems, it does not rely on a signal traveling through the entire depth of the network. This makes it exceptionally robust for extremely deep architectures where traditional backpropagation often fails.
  • Competitive Accuracy: Across the board, pdADMM outperformed standard Gradient Descent, Adagrad, and Adadelta. Notably, while Adam performed well on simpler datasets, pdADMM showed superior performance on more complex tasks like CIFAR-100, suggesting it is better suited for difficult deep-learning challenges.

Conclusion and Future Work

The pdADMM framework represents a significant step toward true model parallelism in deep learning. By breaking the chain of sequential dependencies, it allows for massive speedups that scale with model depth. While currently focused on feed-forward networks, the underlying logic provides a foundation for extending parallel ADMM to more complex structures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Future research will likely focus on minimizing communication overhead between parallel workers to further maximize these hardware-level performance gains.

Summary updated

BibTeX
@inproceedings{wang2020toward,
  author = {Junxiang Wang and Zheng Chai and Yue Cheng and Liang Zhao},
  title = {{Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework}},
  booktitle = {ICDM},
  year = {2020},
  doi = {10.1109/icdm50108.2020.00068},
}
Figure