SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks

Mojtaba Valipour, Mehdi Rezagholizadeh, Hossein Rajabzadeh, Marzieh S. Tahaei, Boxing Chen, Ali Ghodsi

Abstract

Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models. There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g. depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models. To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs. Thanks to a generalized nested architecture (which we refer as sorted architecture) with shared parameters and its novel update scheme combining random sub-model sampling and a new gradient accumulation mechanism, SortedNet enables the training of sub-models simultaneously along with the training of the main model (without any significant extra training or inference overhead), simplifies dynamic model selection, customizes deployment during inference, and reduces the model storage requirement significantly. The versatility and scalability of SortedNet are validated through various architectures and tasks, including LLaMA, BERT, RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating its superiority over existing dynamic training methods. For example, we introduce a novel adaptive self-speculative approach based on sorted-training to accelerate large language models decoding. Moreover, SortedNet is able to train 160 sub-models at once, achieving at least 96% of the original model's performance.

Introduction

In the world of deep learning, "one size fits all" is a myth. A massive Large Language Model (LLM) that runs beautifully on a cluster of A100 GPUs will crawl on a smartphone, while a tiny model optimized for an edge device often lacks the "intelligence" required for complex tasks. Traditionally, engineers solve this by training, storing, and maintaining dozens of different model versions.

SortedNet, a framework developed by researchers at the University of Waterloo and Huawei Noah’s Ark Lab, turns a single neural network into a "many-in-one" modular system. By training a model such that its internal parts are themselves functional sub-models, SortedNet allows a single set of weights to adapt dynamically to any computational budget.

The Challenge of Dynamic Deployment

Deploying DNNs across diverse hardware—from cloud servers to IoT sensors—usually requires model compression or training multiple standalone models. Existing solutions like Once-for-All (OFA) or Early Exit strategies have significant drawbacks:

  1. Complexity: They often require a multi-stage training process or expensive Neural Architecture Search (NAS) to find the best sub-models.
  2. Performance Drops: Smaller sub-models often perform significantly worse than the main model.
  3. Lack of Generality: Many methods only work for specific architectures or specific dimensions (like only changing the width).

The SortedNet Framework

The core innovation of SortedNet is the Sorted Architecture and its accompanying training scheme.

Overview of SortedNet training and inference compared to standard models and other dynamic architectures.
  • The "Sorted" Concept: This is a generalization of nested architectures. In SortedNet, every sub-model is "tied to the origin." For example, a sub-model might consist of the first $k$ layers (depth) and the first $w$ hidden dimensions (width).
  • Stochastic Training: Instead of just training the big model, the algorithm randomly samples a sub-model at each training iteration.
  • Gradient Accumulation: The gradients from these sampled sub-models are accumulated and used to update the main model's parameters. This ensures that the shared weights are optimized to perform well across all possible sizes.
  • Search-Free Deployment: Because the sub-models are "sorted" by their computational cost and accuracy, there is no need for a complex search after training. One simply picks the sub-model that fits the current hardware's latency or memory budget.

Experimental Evaluation

The authors validated SortedNet across a wide range of tasks and architectures, including ResNet and MobileNetV2 on CIFAR-10, BERT and RoBERTa on the GLUE benchmark, and LLaMA-13B on GSM8K.

Key Insights

  • Modularity as an Inductive Bias: By enforcing a "sorted" structure during training, the framework provides an inductive bias that makes DNN modularity useful rather than accidental.
  • Self-Speculative Decoding: SortedNet can accelerate LLM inference by using a smaller sub-model within a LLaMA model to "draft" tokens quickly, which the full model then verifies. This "self-speculation" happens within a single model checkpoint.
  • Scalability: SortedNet successfully trained 160 different sub-models within a single MobileNetV2, a significant jump over previous methods.

Results and Impact

SortedNet sub-models achieved at least 96% of the performance of the original full-sized model. On the GLUE benchmark, Sorted-RoBERTa sub-models outperformed standard baselines by an average of 23%.

Performance landscape of SortedNet sub-models across varying depths and widths, showing high accuracy retention and efficiency trade-offs.

Furthermore, the self-speculative approach achieved a ~2x speedup in inference for LLaMA-13B without losing accuracy. Since all sub-models share the same parameters, storage requirements are reduced to that of a single model, even though it provides the functionality of hundreds.

Conclusion and Future Work

SortedNet offers a versatile and scalable solution for the "many-in-one" model problem. While highly effective for fine-tuning, future work will explore how SortedNet behaves during large-scale, from-scratch pre-training. Additionally, the authors suggest exploring "adaptive exits," where the model decides on-the-fly to stop processing an input at an early layer to further save power.

Summary updated

Abstract

Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models. There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g. depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models. To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs. Thanks to a generalized nested architecture (which we refer as \textit{sorted} architecture in this paper) with shared parameters and its novel update scheme combining random sub-model sampling and a new gradient accumulation mechanism, SortedNet enables the training of sub-models simultaneously along with the training of the main model (without any significant extra training or inference overhead), simplifies dynamic model selection, customizes deployment during inference, and reduces the model storage requirement significantly. The versatility and scalability of SortedNet are validated through various architectures and tasks, including LLaMA, BERT, RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating its superiority over existing dynamic training methods. For example, we introduce a novel adaptive self-speculative approach based on sorted-training to accelerate large language models decoding. Moreover, SortedNet is able to train 160 sub-models at once, achieving at least 96\% of the original model's performance.
BibTeX
@misc{valipour2023sortednet,
  author = {Mojtaba Valipour and Mehdi Rezagholizadeh and Hossein Rajabzadeh and Marzieh S. Tahaei and Boxing Chen and Ali Ghodsi},
  title = {{SortedNet, a Place for Every Network and Every Network in its Place: Towards a Generalized Solution for Training Many-in-One Neural Networks}},
  year = {2023},
  eprint = {2309.00255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00255},
}
Figure