Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models

Han Xiang Choong, Yew-Soon Ong, Abhishek Gupta, Caishun Chen, Ray Lim

TL;DR

This paper introduces the "Set of Sets" concept: a collection of compact, specialized models derived from a single large "Jack of All Trades" (JAT) model. By using a novel evolutionary multitasking approach, they demonstrate that these specialized models can outperform the original JAT on specific tasks while being significantly smaller, making them ideal for resource-constrained environments.

What Problem Does This Paper Address?

The current trend in deep learning is towards massive, general-purpose models (JATs) that can perform a wide array of tasks. While powerful, these JATs are computationally expensive, difficult to deploy on edge devices, and can suffer from inter-task interference, leading to suboptimal performance on individual tasks. The paper addresses the challenge of creating specialized models that retain the benefits of a JAT's pre-training but are efficient and performant for specific use cases, especially in resource-limited scenarios. They aim to bridge the gap between massive, general models and the need for efficient, task-specific solutions.

Methods

The core contribution is the development of a "Set of Sets" approach. This involves:

  • Model Compression via Subnetwork Isolation: Starting with a large pre-trained JAT, the method identifies and extracts compact subnetworks that are specialized for individual tasks. This is achieved by using binary masks to prune parameters.
  • Multi-Objective, Multi-Task Optimization: The optimization process simultaneously considers two objectives: minimizing model size (using the L1-norm of the mask) and minimizing task-specific loss. This is crucial for finding optimal trade-offs between performance and efficiency.
  • One-Pass Neuroevolutionary Multitasking: A novel evolutionary algorithm (MO-MFEA) is employed to evolve the Set of Sets in a single optimization run. This approach leverages inter-task crossover to transfer knowledge between tasks, accelerating convergence and improving efficiency compared to single-task optimization.

Datasets

The paper evaluates their approach on four diverse datasets:

  • Multilingual Translation: WMT-19 dataset, specifically for Czech-to-English and German-to-English translation.
  • Time-Series Regression: Beijing Air Quality dataset, focusing on predicting PM2.5 levels for specific geographical areas.
  • Image Classification: MNIST and CIFAR-10 datasets, with tasks defined as subsets of classes (e.g., even/odd numbers for MNIST, animals/vehicles for CIFAR-10).

Key Insights and Ideas

  • Specialization Trumps Generalization (for specific tasks): Even highly capable JATs can be outperformed by smaller, specialized models derived from them, suggesting that inter-task interference in JATs can hinder peak performance on individual tasks.
  • Efficiency is Achievable: The "Set of Sets" approach demonstrates significant model size reductions (up to ~95.5% in some cases) without sacrificing, and often improving, performance.
  • Evolutionary Multitasking is Key: Performing multi-task optimization in a single evolutionary run is significantly more efficient than training specialized models independently, highlighting the benefits of knowledge transfer between tasks.
  • JATs as Powerful Initialization: Large pre-trained JATs provide a strong foundation, allowing compressed subnetworks to achieve high performance with less training.

Key Results

  • Performance Improvement: In most experiments, the evolved specialized models (MOTs) achieved higher BLEU scores (translation) or lower RMSE (regression) than the original JAT. For example, a specialized model for Beijing air quality prediction achieved an 18.5% RMSE reduction over the JAT.
  • Significant Compression: Models achieved substantial size reductions, with the smallest specialist for CIFAR-10 Animals being ~95.5% smaller than the JAT.
  • Efficiency Gains: The one-pass evolutionary multitasking approach converged significantly faster (demonstrated by hypervolume indicator trends) than single-task optimization, requiring approximately 8 hours for a full run compared to 23 hours for a state-of-the-art single-task method.
  • Broad Applicability: The method was successfully applied across diverse domains (NLP, regression, image classification) and neural architectures (transformers, LSTMs, CNNs).

Limitations and Future Work

  • Fine-tuning is Crucial: While evolutionary optimization uncovers good initializations, fine-tuning with gradient descent is necessary to achieve peak performance.
  • Mask Dimensionality: For very large models, the dimensionality of the binary mask can become a bottleneck, necessitating techniques like parameter grouping.
  • Future Work: The authors suggest further research into analyzing task-specific neural substructures, exploring different evolutionary algorithms, and implementing the Set of Sets concept in real-world edge device scenarios. They also emphasize the need for a comprehensive analysis of the Set of Sets behavior across various task domains and environmental conditions.

Summary updated

Abstract

For deep learning, size is power. Massive neural nets trained on broad data for a spectrum of tasks are at the forefront of artificial intelligence. These large pre-trained models or “Jacks of All Trades” (JATs), when fine-tuned for downstream tasks, are gaining importance in driving deep learning advancements. However, environments with tight resource constraints, changing objectives and intentions, or varied task requirements, could limit the real-world utility of a singular JAT. Hence, in tandem with current trends towards building increasingly large JATs, this paper conducts an initial exploration into concepts underlying the creation of a diverse set of compact machine learning model sets. Composed of many smaller and specialized models, the Set of Sets is formulated to simultaneously fulfil many task settings and environmental conditions. A means to arrive at such a set tractably in one pass of a neuroevolutionary multitasking algorithm is presented for the first time, bringing us closer to models that are collectively “Masters of All Trades”.
BibTeX
@inproceedings{choong2023jack,
  author = {Han Xiang Choong and Yew-Soon Ong and Abhishek Gupta and Caishun Chen and Ray Lim},
  title = {{Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models}},
  booktitle = {IEEE Comput. Intell. Mag},
  year = {2023},
  url = {https://openreview.net/forum?id=ivkTW5ZsRp},
}
Figure