What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition

Carolin Holtermann, Markus Frohmann, Navid Rekabsaz, Anne Lauscher

Abstract

As Large Language Models (LLMs) become increasingly central to natural language processing, the need for efficient domain adaptation grows. While modular adapters offer a solution for specific domains, combining these modules for unseen, zero-shot scenarios remains a challenge. This paper introduces a unified framework to systematically select, weight, and compose these "knowledge packets." Our investigation reveals that simple corpus-based weighting methods often outperform complex model-based alternatives, and that ensembling model outputs consistently yields superior results compared to averaging internal parameters.

Introduction

Pre-trained language models (PLMs) serve as the foundation for modern NLP, yet their static nature limits their effectiveness in specialized domains like legal or medical text. To bridge this gap, researchers utilize adapters—small, modular sets of parameters trained on domain-specific data.

However, a significant challenge arises at inference time: when a model encounters a completely new domain, how should it decide which existing adapters to use and how much influence each should have? This "zero-shot knowledge composition" problem has seen various proposed strategies, but until now, they lacked a fair, unified comparison. This research addresses this gap by evaluating whether we should merge module weights (Parameter Averaging) or average final predictions (Output Ensembling), and which metrics best identify the most relevant modules for a given task.

A Unified Framework for Composition

The researchers propose a three-step framework to standardize the composition process:

  1. Selection: Identifying the top-$k$ most relevant adapters from a library based on the input text.
  2. Weighting: Assigning a numerical importance value ($\omega$) to each selected adapter.
  3. Composition: Combining the adapters using one of two primary methods:
  4. Parameter Averaging: Merging the actual weights of the adapters into a single "model soup."
    • Output Ensembling: Running each adapter-equipped model separately and averaging their resulting probability distributions.

Scoring and Weighting Strategies

To determine adapter relevance, five distinct strategies were tested:

  • Uniform: All selected adapters receive equal weight.
  • Corpus-based (SENTSIM & TF-IDF): These methods measure the similarity between the new text and the adapters' original training data using sentence embeddings or word frequencies.
  • Model-based (PRIOR & ENTROPY): These use Bayesian probability to determine which domain the model "recognizes" the text as belonging to, or measure model uncertainty (entropy) on the new data.

Experimental Setup

The study utilized the C4 dataset, a massive web-crawl collection, divided into:

  • 21 Training Domains: Including diverse sources such as wired.com and instructables.com.
  • 10 Evaluation Domains: Unseen domains used to test zero-shot performance, such as reuters.com, ncbi.nlm.nih.gov (medical), and yelp.com.

The framework was evaluated across different model families, specifically GPT-2 and DeBERTa-base.

Key Insights and Results

The Power of Simplicity

One of the study's most significant findings is that simple corpus-based methods, such as TF-IDF and SENTSIM, frequently outperform more mathematically complex model-based methods. These lexical and semantic similarity measures are not only effective at identifying relevant knowledge but are also significantly more computationally efficient.

Ensembling vs. Averaging

While Parameter Averaging is more efficient during inference (as it only requires a single model pass), it often suffers from performance degradation as more adapters are added. In contrast, Output Ensembling proved to be more robust, consistently yielding lower perplexity across various domains. The research suggests that adding too many modules in parameter space introduces "noise" that can outweigh the benefits of the additional knowledge.

Predictability and Performance

Through Meta-Regression experiments, the researchers demonstrated that zero-shot performance is highly predictable. With correlation scores reaching up to 0.96, it is possible to forecast which adapter combinations will be most effective before execution.

Specific performance gains were notable in the DeBERTa-base model, where the addition of even a single relevant adapter reduced perplexity by massive margins (up to -183,662.70 points), highlighting the critical importance of modular adaptation for certain architectures.

Environmental Impact

The efficiency gap between methods is also environmental. Model-based scoring (like ENTROPY) is computationally heavy, producing up to 61.17g of CO2 in testing, whereas TF-IDF ensembling produced only 3.91g, making the simpler methods more sustainable for large-scale deployment.

Limitations and Future Work

Despite the framework's success, the authors note several areas for further exploration:

  • Metric Scope: The study primarily uses perplexity as the success metric. Future research should evaluate performance on downstream tasks like summarization or sentiment analysis.
  • Dataset and Language: The current scope is limited to the English-language C4 dataset; results may vary on non-English or more specialized corpora.
  • Architectural Variety: While tested on adapters, the framework could be extended to other modular approaches like LoRA or Mixture-of-Experts (MoE).
  • The $k$ Problem: Automatically determining the optimal number of adapters ($k$) for any given input remains an open challenge, though "early stopping" based on weight distribution shows promise.

Conclusion

This unified framework clarifies the landscape of zero-shot knowledge composition. By demonstrating that simple, corpus-based weighting and output ensembling provide a winning combination of performance and efficiency, the researchers provide a clear roadmap for more modular and adaptable language models.

Summary updated

Abstract

The knowledge encapsulated in a model is the core factor determining its final performance on downstream tasks. Much research in NLP has focused on efficient methods for storing and adapting different types of knowledge, e.g., in dedicated modularized structures, and on how to effectively combine these, e.g., by learning additional parameters. However, given the many possible options, a thorough understanding of the mechanisms involved in these compositions is missing, and hence it remains unclear which strategies to utilize. To address this research gap, we propose a novel framework for zero-shot module composition, which encompasses existing and some novel variations for selecting, weighting, and combining parameter modules under a single unified notion. Focusing on the scenario of domain knowledge and adapter layers, our framework provides a systematic unification of concepts, allowing us to conduct the first comprehensive benchmarking study of various zero-shot knowledge composition strategies. In particular, we test two module combination methods and five selection and weighting strategies for their effectiveness and efficiency in an extensive experimental setup. Our results highlight the efficacy of ensembling but also hint at the power of simple though often-ignored weighting methods. Further in-depth analyses allow us to understand the role of weighting vs. top-k selection, and show that, to a certain extent, the performance of adapter composition can even be predicted.
BibTeX
@inproceedings{holtermann2024what,
  author = {Carolin Holtermann and Markus Frohmann and Navid Rekabsaz and Anne Lauscher},
  title = {{What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition}},
  booktitle = {EACL (Findings)},
  year = {2024},
}
Figure