Abstract: Alternating Direction Method of Multipliers (ADMM) has recently been proposed as a potential alternative optimizer to the Stochastic Gradient Descent (SGD) for deep learning problems. This is because ADMM can solve gradient vanishing and poor conditioning problems. Moreover, it has shown good scalability in many large-scale deep learning applications. However, there still lacks a parallel ADMM computational framework for deep neural networks because of layer dependency among variables. In this paper, we propose a novel parallel deep learning ADMM framework (pdADMM) to achieve layer parallelism: parameters in each layer of neural networks can be updated independently in parallel. The convergence of the proposed pdADMM to a critical point is theoretically proven under mild conditions. The convergence rate of the pdADMM is proven to be $o(1/k)$ where $k$ is the number of iterations. Extensive experiments on six benchmark datasets demonstrated that our proposed pdADMM can lead to more than 10 times speedup for training large-scale deep neural networks, and outperformed most of the comparison methods.
Introduction: Breaking the Sequential Bottleneck
Standard deep learning training relies heavily on Stochastic Gradient Descent (SGD) and the backpropagation algorithm. While these methods have driven the current AI revolution, they suffer from a fundamental structural flaw: inherent sequentiality. To calculate gradients for any given layer, the system must wait for the results from the subsequent layer. This "locking" mechanism prevents true model parallelism, where different computing units could ideally process different layers of a deep network simultaneously.
Furthermore, gradient-based methods are frequently plagued by the "vanishing gradient" problem in very deep architectures and "poor conditioning," where minor input variations lead to unstable gradient swings. While the Alternating Direction Method of Multipliers (ADMM) has been explored as a gradient-free alternative to mitigate these issues, previous implementations still required layers to be updated one after another. This research introduces pdADMM, a framework designed to finally decouple these layer dependencies.
The pdADMM Framework
The core innovation of pdADMM lies in a mathematical reformulation of the feed-forward neural network optimization problem. By treating the connections between layers as constraints rather than direct functional dependencies, the authors enable simultaneous updates across the entire model architecture.
Key Methodological Steps:
- Variable Splitting: The connection between layers is split into two independent variables ($q_l$ for output and $p_{l+1}$ for input). The physical connection is then treated as a mathematical constraint ($p_{l+1} = q_l$).
- Augmented Lagrangian: Using this method, the global training objective is decomposed into smaller, independent subproblems—one for each layer.
- Layer Parallelism: Because these subproblems are decoupled, different workers (such as separate GPUs or CPU cores) can solve them at the same time. A worker can update Layer 1 while another simultaneously updates Layer 10.
- Quadratic Approximation: To maintain efficiency and avoid expensive matrix inversions, the framework employs a quadratic approximation technique to simplify weight and input updates.
Experimental Evaluation and Results
The framework was rigorously tested on six image classification benchmarks: MNIST, Fashion MNIST, kMNIST, SVHN, CIFAR-10, and CIFAR-100. The results highlight a significant leap in training efficiency and scalability.
Performance Insights:
- Linear Speedup Scaling: As shown in the experimental data, the speedup is not static; it scales linearly with the depth of the network. In tests with 19 hidden layers, the system achieved over 11x speedup, proving that the deeper the network, the greater the benefit of the pdADMM parallelization.
- Immunity to Vanishing Gradients: Since pdADMM is gradient-free and solves local subproblems, it does not rely on a signal traveling through the entire depth of the network. This makes it exceptionally robust for extremely deep architectures where traditional backpropagation often fails.
- Competitive Accuracy: Across the board, pdADMM outperformed standard Gradient Descent, Adagrad, and Adadelta. Notably, while Adam performed well on simpler datasets, pdADMM showed superior performance on more complex tasks like CIFAR-100, suggesting it is better suited for difficult deep-learning challenges.
Conclusion and Future Work
The pdADMM framework represents a significant step toward true model parallelism in deep learning. By breaking the chain of sequential dependencies, it allows for massive speedups that scale with model depth. While currently focused on feed-forward networks, the underlying logic provides a foundation for extending parallel ADMM to more complex structures like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). Future research will likely focus on minimizing communication overhead between parallel workers to further maximize these hardware-level performance gains.
Summary updated