What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition

Carolin Holtermann, Markus Frohmann, Navid Rekabsaz, Anne Lauscher

TL;DR

Researchers have introduced a unified framework to systematically select, weight, and combine modular "knowledge adapters" for zero-shot domain adaptation. Their benchmarking reveals that while output ensembling generally beats parameter averaging, surprisingly simple and efficient corpus-based methods like TF-IDF often outperform complex model-based strategies.

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) relies on massive Pre-trained Language Models (PLMs). To make these models specialized without retraining the whole thing, researchers use adapters—small, modular parameter blocks trained for specific tasks or domains.

The challenge arises at inference time: if you encounter a new, unseen domain (zero-shot), how do you decide which existing adapters to use and how to combine them? While many strategies exist—like "model soups" (averaging parameters) or ensembling outputs—there hasn't been a unified way to compare them. We didn't know if we should pick the top-3 most similar adapters, weight them all equally, or use complex Bayesian priors to determine their importance.

Methods

The authors propose a three-step unified framework for on-demand module composition, illustrated in Figure 1:

Figure 1: Our unified framework for on-demand module composition consisting of three steps: selection, weight- ing, and final combination. We show the example of zero-shot domain adaptation with adapter layers.
  1. Module Selection: Identifying the $k$ most suitable adapters from a library.
  2. Weighting: Assigning a numerical importance ($\omega_i$) to each selected adapter.
  3. Composition: The final combination method. The paper compares two main approaches:
  4. Parameter Averaging: Merging the weights of the adapters into a single set of parameters (efficient at inference).
  5. Output Ensembling: Running the input through each adapter separately and averaging their predictions (more computationally expensive).

To drive selection and weighting, they tested five Scoring Strategies:

  • Uniform: Equal weights for all.
  • Semantic Sentence Similarity (SentSim): Using Sentence-BERT to find domains with similar meanings.
  • TF-IDF: A classic information retrieval metric that looks at lexical (word-level) overlap.
  • Domain Prior: A Bayesian approach estimating the probability that a sequence belongs to a specific source domain.
  • Entropy: Measuring model uncertainty; the adapter with the lowest uncertainty on new data gets the highest weight.

Datasets

The study utilized a diverse set of 31 domains primarily sourced from the C4 dataset (a massive crawl of the web), categorized by their source website (e.g., wired.com, dailymail.co.uk).

  • Training: 21 domains used to train individual adapters.
  • Evaluation: 10 unseen domains (e.g., reuters.com, techcrunch.com, and the yelp.com dataset) to test zero-shot adaptation performance.

Key Insights and Ideas

  • Ensembling is the Gold Standard (Mostly): Across almost all tests, ensembling the outputs of adapters yielded lower perplexity (better performance) than averaging their parameters.
  • Simple is Sophisticated: One of the most surprising findings was that TF-IDF and SentSim—simple methods based on text overlap—often outperformed "smarter" model-based methods like Entropy or Domain Priors. They are also significantly cheaper to compute.
  • The "More is Less" Trap: Adding more adapters ($k$) doesn't always help. For some models, performance peaks at a small number of adapters and then degrades as more (potentially noisy or irrelevant) modules are added.
  • Performance is Predictable: By using meta-regression, the authors showed they could predict how well a specific combination of adapters would perform on a new domain based on metadata, potentially saving massive amounts of experimental time.

Key Results

The quantitative results highlight a clear trade-off between performance and efficiency. As shown in Figure 2, ensembling (dashed lines) consistently achieves lower perplexity than averaging (solid lines) as more adapters are added.

Figure 2: Performance comparison between Averaging and Ensembling across different numbers of adapters.

However, this performance comes at a cost. Figure 5 demonstrates the environmental impact: ensembling causes a linear increase in $CO_2$ emissions because the model must perform a full forward pass for every adapter in the ensemble. In contrast, parameter averaging maintains a flat emission profile regardless of how many adapters are merged.

Figure 5: Efficiency and CO2 emissions of different strategies.

Specifically, for the gpt2-base model, ensembling with TF-IDF selection reached a perplexity of ~22.2, while the best averaging method was slightly higher at ~22.5. While the gap seems small, in the world of language modeling, these are significant gains for "free" zero-shot adaptation.

Limitations and Future Work

  • Domain Definition: The authors defined domains by their source website. In reality, domains are "fuzzy" and can overlap significantly.
  • Metric Proxy: The study relies on perplexity as the primary evaluation metric. While standard for language models, it doesn't always perfectly correlate with performance on specific downstream tasks like summarization or sentiment analysis.
  • Module Variety: The research focused exclusively on adapters. Future work could extend this framework to other modular architectures like Mixture-of-Experts (MoE) or LoRA.
  • Scaling: While they tested GPT-2 Large, the behavior of these composition strategies on truly massive models (70B+ parameters) remains an open question.

Summary updated

Abstract

The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.
BibTeX
@inproceedings{holtermann2024what,
  author = {Carolin Holtermann and Markus Frohmann and Navid Rekabsaz and Anne Lauscher},
  title = {{What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition}},
  booktitle = {EACL (Findings)},
  year = {2024},
}
Figure