Multi-Objective Optimization for Sparse Deep Neural Network Training

Sèdjro S. Hotegni, Sebastian Peitz, Manuel BerkemeierarXiv 2023 · arXiv · OpenReview

Abstract

Different conflicting optimization criteria arise naturally in various Deep Learning scenarios. These can address different main tasks (i.e., in the setting of Multi-Task Learning), but also main and secondary tasks such as loss minimization versus sparsity. The usual approach is a simple weighting of the criteria, which formally only works in the convex setting. In this paper, we present a Multi-Objective Optimization algorithm using a modified Weighted Chebyshev scalarization for training Deep Neural Networks (DNNs) with respect to several tasks. By employing this scalarization technique, the algorithm can identify all optimal solutions of the original problem while reducing its complexity to a sequence of single-objective problems. The simplified problems are then solved using an Augmented Lagrangian method, enabling the use of popular optimization techniques such as Adam and Stochastic Gradient Descent, while efficaciously handling constraints. Our work aims to address the (economical and also ecological) sustainability issue of DNN models, with a particular focus on Deep Multi-Task models, which are typically designed with a very large number of weights to perform equally well on multiple tasks. Through experiments conducted on two Machine Learning datasets, we demonstrate the possibility of adaptively sparsifying the model during training without significantly impacting its performance, if we are willing to apply task-specific adaptations to the network weights.

Introduction

As Deep Learning models grow in size, they become increasingly expensive to train and deploy, raising both economic and ecological sustainability concerns. This is particularly true in Multi-Task Learning (MTL), where a single model is expected to perform well on several different tasks simultaneously.

The challenge is two-fold:

  1. Conflicting Objectives: Improving performance on Task A might degrade Task B, and reducing model size (sparsity) usually comes at the cost of accuracy.
  2. Non-Convexity: The standard way to handle multiple goals is to use a "weighted sum" of losses. However, this only works if the trade-off space (the Pareto front) is convex. In deep learning, these spaces are often non-convex, meaning standard weighting schemes miss the best possible "compromise" solutions.

This paper introduces a novel framework for training sparse Deep Multi-Task Learning (MTL) models by treating accuracy and model size as a non-convex Multi-Objective Optimization (MOO) problem. By combining a modified Weighted Chebyshev scalarization with a new "Monitored" network architecture, the authors demonstrate that you can significantly prune and compress models—sometimes by over 50%—while actually improving performance across multiple tasks.

Methods

The proposed methodology moves beyond simple loss weighting by integrating several advanced optimization and architectural techniques:

  • Modified Weighted Chebyshev (WC) Scalarization: This technique allows the optimizer to identify any optimal solution on the Pareto front, regardless of whether the space is convex or non-convex.
  • Augmented Lagrangian (AL) Method: To make the complex WC objective solvable with standard tools like Adam or SGD, the constrained MOO problem is transformed into a sequence of unconstrained problems.
  • GrOWL Regularization: Group Ordered Weighted $l_1$ (GrOWL) is used to promote both sparsity (zeroing out neurons) and parameter tying (forcing groups of neurons to share the same values), leading to high compression rates.
  • MDMTN Architecture: The Monitored Deep Multi-Task Network (MDMTN) uses small, task-specific "monitors" to capture nuances the shared backbone might miss, without significantly increasing the parameter count.
The Monitored Deep Multi-Task Network (MDMTN) architecture.

Datasets

The approach was evaluated on two benchmarks:

  1. MultiMNIST: A standard MTL dataset where two handwritten digits are glued together with overlap.
  2. Cifar10Mnist: A newly introduced dataset created by the authors. It overlays padded MNIST digits onto CIFAR-10 images. This is significantly more challenging than MultiMNIST because it forces the model to learn features from two very different data distributions: natural images and handwritten strokes.
Samples from the Cifar10Mnist dataset, overlaying MNIST digits on CIFAR-10 images.

Key Results and Insights

The experiments yielded several significant findings regarding the trade-off between model complexity and performance:

  • Sparsity as a Performance Booster: On the MultiMNIST dataset, the sparsified model (57% of neurons removed) achieved 96.85% average accuracy, outperforming the dense baseline. This suggests that multi-objective pruning can act as a powerful regularizer, stripping away redundant noise that hinders multi-task generalization.
  • The "Monitor" Advantage: The MDMTN architecture consistently outperformed standard Hard Parameter Sharing. The small task-specific monitors provide just enough flexibility to resolve task conflicts without the massive memory overhead of "soft sharing" approaches.
  • Visualizing the Trade-off: By using WC scalarization, the authors mapped out the 3D Pareto front (Task 1 vs. Task 2 vs. Sparsity). This allows a developer to choose exactly how much accuracy they are willing to trade for a smaller deployment footprint.
  • Efficiency: Despite the complexity of the optimization, the method typically demonstrated shorter training times than alternative MTL approaches like KDMTL or MTAN. On Cifar10Mnist, the model achieved 77.65% average accuracy with 29.31% sparsity.

Conclusion

This work demonstrates that multi-objective optimization can effectively balance task performance and model sparsity in deep learning. However, when tasks are "significantly dissimilar" (as in the Cifar10Mnist case), the relationship between sparsity and performance becomes more volatile. Future work will focus on developing more adaptive mechanisms to handle these highly dissimilar tasks and exploring how to further automate the selection of preference weights within the MOO framework.

Code is available at https://github.com/salomonhotegni/MDMTN

Summary updated

Abstract

Different conflicting optimization criteria arise naturally in various Deep Learning scenarios. These can address different main tasks (i.e., in the setting of Multi-Task Learning), but also main and secondary tasks such as loss minimization versus sparsity. The usual approach is a simple weighting of the criteria, which formally only works in the convex setting. In this paper, we present a Multi-Objective Optimization algorithm using a modified Weighted Chebyshev scalarization for training Deep Neural Networks (DNNs) with respect to several tasks. By employing this scalarization technique, the algorithm can identify all optimal solutions of the original problem while reducing its complexity to a sequence of single-objective problems. The simplified problems are then solved using an Augmented Lagrangian method, enabling the use of popular optimization techniques such as Adam and Stochastic Gradient Descent, while efficaciously handling constraints. Our work aims to address the (economical and also ecological) sustainability issue of DNN models, with a particular focus on Deep Multi-Task models, which are typically designed with a very large number of weights to perform equally well on multiple tasks. Through experiments conducted on two Machine Learning datasets, we demonstrate the possibility of adaptively sparsifying the model during training without significantly impacting its performance, if we are willing to apply task-specific adaptations to the network weights. Code is available at https://github.com/salomonhotegni/MDMTN
BibTeX
@misc{hotegni2023multi,
  author = {Sèdjro S. Hotegni and Sebastian Peitz and Manuel Berkemeier},
  title = {{Multi-Objective Optimization for Sparse Deep Neural Network Training}},
  year = {2023},
  eprint = {2308.12243},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12243},
}
Figure