Task-Agnostic Experts Composition for Continual Learning

Luigi Quarantiello, Andrea Cossu, Vincenzo LomonacoarXiv 2025 · DOI · arXiv

Introduction

Modern artificial intelligence has trended toward "foundation models"—massive, monolithic architectures with billions of parameters. While powerful, these models are computationally expensive to train and prone to "catastrophic forgetting" in continual learning (CL) scenarios, where new knowledge often overwrites previously learned information. This paper introduces Expert Composition (EC), a modular alternative that leverages specialized, small-scale models to achieve superior performance with significantly lower resource requirements.

The Expert Composition Framework

The core philosophy of EC is compositionality: the ability to understand complex scenes by decomposing them into simpler, familiar parts. Instead of one giant network, the authors propose a two-stage pipeline.

1. Expert Training

The system begins by training a library of specialized "experts." In this study, seven experts were developed using the ResNet-18 architecture. Each expert is trained on a narrow subset of data—specifically, three object classes from the GQA (Genome Question Answering) dataset. A critical component of this training is the inclusion of a generic "other" class. This allows an expert to recognize when an input falls outside its specific domain, effectively acting as a filter during the composition phase.

2. Zero-Shot Composition

To process complex, multi-object images, the EC framework employs a zero-shot approach. When presented with a composite image (such as a 2x2 grid from the CGQA benchmark), the system extracts quadrants and passes them through the expert library. Each expert provides a prediction, and these individual "votes" are concatenated to form the final multi-label output.

The Expert Composition (EC) workflow: specialized experts are trained independently and then composed zero-shot to perform multi-label classification on complex scenes.

Because the experts are never retrained on the composite tasks, they are immune to catastrophic forgetting. This modularity ensures that the system retains its specialized knowledge indefinitely.

Experimental Results

The researchers evaluated EC against state-of-the-art CL baselines using the Compositional GQA (CGQA) benchmark. CGQA specifically tests a model's ability to recognize novel combinations of objects it has seen individually.

Performance and Accuracy

The EC method achieved a remarkable 58% accuracy on the CGQA benchmark. This nearly doubles the performance of traditional CL algorithms like Experience Replay (ER), which reached only 26%. Other common baselines, such as Elastic Weight Consolidation (EWC) and Learning without Forgetting (LwF), struggled significantly, with accuracies ranging between 10% and 20%.

Computational Efficiency

Beyond accuracy, the EC approach offers drastic improvements in sustainability and speed. Training a single expert required approximately 13 minutes on an NVIDIA Tesla T4 GPU. In contrast, standard monolithic CL baselines required nearly 4 hours of training time. Furthermore, because EC is task-agnostic and zero-shot, it achieves peak performance from the very first "experience," eliminating the need for the extensive data exposure required by traditional models.

Conclusion

Expert Composition represents a shift away from the "bigger is better" paradigm in AI. By prioritizing modularity and specialized knowledge, the authors demonstrate that smaller, task-agnostic models can be composed to solve complex problems more effectively than monolithic structures. While currently tested on synthetic datasets, the success of EC suggests a promising path toward more efficient, sustainable, and robust continual learning systems. Future research will focus on scaling this approach to more realistic visual environments and automating the selection of experts from large-scale model repositories.

Summary updated

Abstract

Compositionality is one of the fundamental abilities of the human reasoning process, that allows to decompose a complex problem into simpler elements. Such property is crucial also for neural networks, especially when aiming for a more efficient and sustainable AI framework. We propose a compositional approach by ensembling zero-shot a set of expert models, assessing our methodology using a challenging benchmark, designed to test compositionality capabilities. We show that our Expert Composition method is able to achieve a much higher accuracy than baseline algorithms while requiring less computational resources, hence being more efficient.
BibTeX
@misc{quarantiello2025task,
  author = {Luigi Quarantiello and Andrea Cossu and Vincenzo Lomonaco},
  title = {{Task-Agnostic Experts Composition for Continual Learning}},
  year = {2025},
  eprint = {2506.15566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.15566},
}
Figure