Jack and Masters of All Trades: One-Pass Learning of a Set of Model Sets from Foundation Models

Han Xiang Choong, Yew-Soon Ong, Abhishek Gupta, Ray Lim

TL;DR

This paper introduces the "Set of Sets" concept: a collection of specialized, compact models derived from a single large "Jack of All Trades" (JAT) model. By using a novel one-pass evolutionary multitasking approach, they demonstrate how to efficiently create these specialized models, achieving better performance and significantly smaller sizes than the original JAT, making AI more deployable in resource-constrained environments.

What Problem Does This Paper Address?

The current trend in AI is towards massive, general-purpose "Jacks of All Trades" (JATs) like GPT-3 or ViT-G/14. While powerful, these models are computationally expensive and often overkill for specific tasks. Deploying them in resource-constrained environments (like mobile devices or edge computing) is challenging due to their size and energy demands. Furthermore, a single JAT might not perform optimally on every specific task due to internal conflicts or inefficiencies from trying to be a master of too many domains. This paper addresses the need for a more efficient and adaptable approach to AI deployment by creating a diverse set of smaller, specialized models that collectively act as "Masters of All Trades" (MATs).

Methods

The core innovation is the "Set of Sets" approach, which leverages a large pre-trained JAT as a foundation. Instead of training specialized models from scratch, they "prune" and adapt parts of the JAT. This is achieved through a novel one-pass neuroevolutionary multitasking algorithm. This algorithm simultaneously optimizes for multiple tasks and multiple objectives (performance and model size). It uses a binary mask to select which parts of the JAT to retain for each specialized model (MOT - Master of One Trade). The key novelty lies in the multi-objective, multi-task optimization formulation and its implementation using an evolutionary multitasking algorithm, which allows for efficient transfer of learned knowledge between tasks during the optimization process. This contrasts with traditional methods that would train each specialized model independently.

Datasets

The paper evaluates their approach on four diverse datasets across different domains:

  • Multilingual Translation: WMT-19 dataset, specifically for Czech-to-English and German-to-English translation.
  • Time-Series Regression: Beijing Air Quality dataset, focusing on predicting PM2.5 levels for specific geographical areas.
  • Image Classification: MNIST and CIFAR-10 datasets, with tasks involving subsets of classes (e.g., even vs. odd numbers for MNIST, animals vs. vehicles for CIFAR-10).

The JATs used were pre-trained models: Facebook's M2M100-418M for translation, a bidirectional LSTM for air quality, and ResNet-18 for image classification.

Key Insights and Ideas

  • Specialization Trumps Generalization (for specific tasks): Even though JATs are powerful, specialized models (MOTs) derived from them can often outperform the JAT on their specific tasks, especially when inter-task interference in the JAT is removed.
  • Compression is Key for Deployment: The ability to significantly reduce model size while maintaining or improving performance is crucial for deploying AI in resource-constrained environments.
  • One-Pass Evolutionary Multitasking is Efficient: Performing multi-task and multi-objective optimization in a single evolutionary run is significantly more efficient than training individual specialized models, leading to faster convergence and better overall results.
  • JATs as Foundation Models for Specialization: Large pre-trained JATs provide an excellent starting point for creating specialized models, offering a strong initialization that accelerates learning and improves performance.
  • The "Set of Sets" as a Practical MAT: A collection of specialized, compact models can effectively act as a "Master of All Trades" by collectively covering a wide range of tasks and environmental conditions, offering a practical alternative to monolithic JATs.

Key Results

  • Performance Improvement: In most cases, the evolved MOTs achieved higher BLEU scores (translation) or lower RMSE (regression) than the original JAT. For example, in the Beijing Air Quality regression task, one specialist model achieved an RMSE reduction of ~18.5% over the JAT.
  • Significant Model Compression: The Set of Sets achieved substantial size reductions. For instance, specialists for CIFAR-10 Animals were reduced by ~95.5%, and for German-to-English translation by ~52%.
  • Efficiency Gains: The one-pass evolutionary multitasking approach was significantly faster, with MO-MFEA runs taking about 8 hours compared to ~23 hours for a single-task evolutionary method (EMOMC) on a similar problem.
  • Hypervolume Improvement: Multi-task optimization led to significantly faster convergence of hypervolumes (a measure of Pareto front quality) compared to single-task optimization.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that fine-tuning is often necessary after the evolutionary process to achieve optimal performance, suggesting that pure pruning might not be sufficient on its own.
  • While performance improvements were seen, in some cases (like CIFAR-10 Vehicles), the JAT was already near-optimal, making further gains difficult.
  • The paper focuses on binary masks for pruning, which is a specific form of compression.
  • Future Work:
    • The authors suggest exploring the behavior of the Set of Sets across a wider variety of task domains, data distributions, and user intentions.
  • Investigating different evolutionary algorithms for generating the Set of Sets is also proposed.
  • The ultimate goal is to demonstrate the utility of this concept through implementation on real-world, resource-constrained edge devices.

Summary updated

Abstract

For deep learning, size is power. Massive neural nets trained on broad data for a spectrum of tasks are at the forefront of artificial intelligence. These large pre-trained models or Jacks of All Trades (JATs), when fine-tuned for downstream tasks, are gaining importance in driving deep learning advancements. However, environments with tight resource constraints, changing objectives and intentions, or varied task requirements, could limit the real-world utility of a singular JAT. Hence, in tandem with current trends towards building increasingly large JATs, this paper conducts an initial exploration into concepts underlying the creation of a diverse set of compact machine learning model sets. Composed of many smaller and specialized models, the Set of Sets is formulated to simultaneously fulfil many task settings and environmental conditions. A means to arrive at such a set tractably in one pass of a neuroevolutionary multitasking algorithm is presented for the first time, bringing us closer to models that are collectively Masters of All Trades.
BibTeX
@inproceedings{choong2023jack,
  author = {Han Xiang Choong and Yew-Soon Ong and Abhishek Gupta and Ray Lim},
  title = {{Jack and Masters of All Trades: One-Pass Learning of a Set of Model Sets from Foundation Models}},
  booktitle = {IEEE Comput. Intell. Mag},
  year = {2023},
  url = {https://openreview.net/forum?id=ivkTW5ZsRp},
}
Figure