SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks

Mojtaba Valipour, Mehdi Rezagholizadeh, Hossein Rajabzadeh, Marzieh S. Tahaei, Boxing Chen, Ali Ghodsi, Ghodsi, Ali

TL;DR

This paper introduces SortedNet, a novel framework for training a single neural network that can efficiently serve multiple sub-models with varying computational budgets. By leveraging a "sorted" architecture and a unique training scheme, SortedNet enables simultaneous training of numerous sub-models without significant overhead, offering a scalable and generalized solution for dynamic model deployment.

What Problem Does This Paper Address?

In the era of deep learning, users often require models tailored to specific performance needs and computational budgets. The traditional approach of training and storing numerous individual models for each user or task is prohibitively expensive and complex. While existing "many-in-one" solutions aim to address this, they often suffer from performance degradation, limited generalization across different model architectures or dimensions (like depth and width), and require extensive model search during training or inference. This paper tackles the challenge of efficiently training and deploying a single model that can dynamically adapt to diverse user requirements without compromising performance.

Methods

SortedNet introduces a "sorted architecture," a generalization of nested architectures, where sub-models are organized in a consistent, origin-tied manner across dimensions like depth and width. This structure, combined with shared parameters, promotes regular knowledge learning. The core innovation lies in its training paradigm:

  • Stochastic Training: Instead of training all sub-models simultaneously or sequentially, SortedNet uses a novel update scheme. At each training iteration, it randomly samples a sub-model from the sorted architecture and updates its parameters.
  • Gradient Accumulation: This technique is employed to effectively train multiple sub-models within a single training step, allowing for more stable and efficient learning across the diverse sub-model set.
  • Search-Free Extraction: The sorted architecture and training process eliminate the need for extensive model search during inference, allowing for efficient selection of the best sub-model for a given budget.

Datasets

The paper evaluates SortedNet on several standard datasets across different domains:

  • Image Classification: CIFAR-10 (used for experiments with ResNet and MobileNet architectures).
  • Natural Language Processing (NLP): GLUE benchmark (used for BERT and RoBERTa models), and GSM8K (used for LLaMA large language models).

Key Insights and Ideas

  • Harnessing Modularity: The paper effectively demonstrates how to leverage the inherent modularity of DNNs (e.g., layers, blocks) to create a unified training framework.
  • Scalability Beyond Depth/Width: SortedNet's "sorted architecture" is shown to be generalizable beyond just depth and width, extending to other dimensions like attention heads.
  • Efficient Training of Many Sub-models: The stochastic sampling and gradient accumulation approach allows for training a large number of sub-models (up to 160) simultaneously without prohibitive computational cost.
  • "Anytime" Networks: The architecture naturally supports "anytime" inference, where a model can produce an output at any stage of its computation, offering flexibility.
  • Theoretical Justification: The paper provides theoretical analysis on parameter convergence and performance bounds, explaining why SortedNet works.

Key Results

  • High Performance Preservation: SortedNet achieved at least 96% of the original model's performance when training up to 160 sub-models on CIFAR-10. Across width, it preserved up to 99% of the largest network's performance.
  • NLP Task Success: Demonstrated strong performance on BERT, RoBERTa, and LLaMA models across various NLP tasks.
  • LLaMA Decoding Acceleration: Introduced an adaptive self-speculative approach based on SortedNet to accelerate LLaMA large language model decoding, achieving up to 2x inference speedup.
  • Reduced Storage: The many-in-one nature significantly reduces model storage requirements compared to training individual models.
  • Generalizability: Validated across diverse architectures (LLaMA, BERT, RoBERTa, ResNet, MobileNet) and tasks.

Limitations and Future Work

  • Pre-training Impact: The paper notes that applying SortedNet during pre-training, rather than fine-tuning, could yield even better results, especially for large models like BERT. This is left for future research.
  • Neuron Dependency Analysis: While the paper explores the impact of sorting on neuron dependency, further investigation into this aspect and its implications is suggested.
  • Classifier Adjustment: While the shared classifier performs well, further adjustments to individual classifiers could potentially yield even better performance for smaller sub-models.
  • Complex Dimensions: While extending to attention heads is mentioned, further exploration of sorting across more complex and combined dimensions could be beneficial.

Summary updated

Abstract

Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models. There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g. depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models. To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs. Thanks to a generalized nested architecture (which we refer as \textit{sorted} architecture in this paper) with shared parameters and its novel update scheme combining random sub-model sampling and a new gradient accumulation mechanism, SortedNet enables the training of sub-models simultaneously along with the training of the main model (without any significant extra training or inference overhead), simplifies dynamic model selection, customizes deployment during inference, and reduces the model storage requirement significantly. The versatility and scalability of SortedNet are validated through various architectures and tasks, including LLaMA, BERT, RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating its superiority over existing dynamic training methods. For example, we introduce a novel adaptive self-speculative approach based on sorted-training to accelerate large language models decoding. Moreover, SortedNet is able to train 160 sub-models at once, achieving at least 96\% of the original model's performance.
BibTeX
@misc{valipour2023sortednet,
  author = {Mojtaba Valipour and Mehdi Rezagholizadeh and Hossein Rajabzadeh and Marzieh S. Tahaei and Boxing Chen and Ali Ghodsi and Ghodsi, Ali},
  title = {{SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks}},
  year = {2023},
  eprint = {2309.00255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00255},
}
Figure