Mixed-effects transformers for hierarchical adaptation

Julia White, Noah D. Goodman, Robert X. D. Hawkins

Abstract

Language differs dramatically from context to context. To some degree, large language models like GPT-3 account for such variation by conditioning on strings of initial input text, or prompts. However, prompting can be ineffective when contexts are sparse, out-of-sample, or extra-textual. In this paper, we introduce the mixed-effects transformer (MET), a novel approach for learning hierarchically-structured prefixes— lightweight modules prepended to an input sequence— to account for structured variation in language use. Specifically, we show how the popular class of mixed-effects regression models may be extended to transformer-based architectures using a regularized prefix-tuning procedure with dropout. We evaluate this approach on several domain-adaptation benchmarks, finding that it learns contextual variation from minimal data while generalizing well to unseen contexts.

Introduction

Language use is inherently heterogeneous, varying significantly across speakers, genres, and time periods. While Large Language Models (LLMs) can adapt via in-context learning (prompting), this mechanism often fails when data is sparse, context is extra-textual (e.g., metadata like User IDs), or when generalizing to out-of-distribution domains.

To address these challenges, we present the Mixed-Effects Transformer (MET). Presented at EMNLP 2022, MET bridges classical statistical mixed-effects modeling with modern transformer architectures. By regularizing domain-specific "prefixes" toward a global average, MET enables models to be both highly specialized to specific contexts and robust enough to handle entirely new ones.

Methodology: The Mixed-Effects Transformer

The MET architecture builds upon prefix-tuning, a parameter-efficient adaptation method where the primary transformer weights remain frozen. Instead of fine-tuning the entire model, MET trains a small set of "virtual tokens" or prefixes that are prepended to the input sequence.

Architecture of the Mixed-effects Transformer (MET) showing hierarchical prefix-tuning.

MET structures these prefixes using the logic of mixed-effects regression:

  • Fixed Effects: The model learns specific parameters for frequently occurring contexts, such as a prefix for the "Horror" genre.
  • Random Effects: To prevent overfitting, specific prefixes are assumed to be drawn from a shared distribution.

A Multi-Layer Perceptron (MLP) maps contextual features to these prefixes. During training, a dropout-based regularization is applied: with a 10% probability, a feature-specific prefix is replaced by a "global" shared prefix ($p^*$). Additionally, a penalty term pulls feature-specific prefixes toward this global mean, forcing the model to share statistical strength across domains while allowing for niche-specific deviations.

Experimental Setup

We evaluated MET across four diverse datasets to test its adaptation and generalization capabilities:

  • Amazon Product Reviews: 100,000 sentences across 10 categories (e.g., Electronics, Beauty).
  • Reddit Corpus: 100,000 sentences from 10 distinct subreddits.
  • Cornell Movie-Dialogs: 10,000 sentences across 10 genres.
  • C4 (Common Crawl): Used for benchmarking against hierarchical adapters across 10 web domains.

Results and Discussion

MET consistently outperforms standard adaptation techniques, particularly in low-data regimes. By providing a "Goldilocks" middle ground between complete pooling (one model for all) and no pooling (separate models for each domain), MET effectively shares statistical strength.

Log perplexity across three corpora as a function of training samples per feature.

Key Findings:

  • Data Efficiency: As shown in the results, MET is remarkably effective when data is scarce. With only 10 to 100 samples per feature, MET significantly outperforms standard prefix-tuning and conditional fine-tuning.
  • Lower Perplexity: MET achieved the lowest log perplexity across all datasets. For instance, on the Movie Corpus, MET scored 3.47 on seen contexts, compared to 3.54 for standard prefix-tuning.
  • Generalization: On unseen contexts, MET significantly outperformed "No Pooling" baselines, which typically fail on new data.
  • Latent Structure: t-SNE visualizations of learned prefixes revealed that MET naturally clusters categories (e.g., grouping "Apparel" product prefixes), indicating it successfully captures underlying data structures.

Conclusion and Future Work

The Mixed-Effects Transformer offers a robust framework for hierarchical domain adaptation. While highly effective, scaling to millions of unique features (like individual User IDs) remains a challenge due to GPU memory constraints. Future research will explore more compact MLP parameterizations and the impact of batch composition on learning the central tendency of the data.

Summary updated

Abstract

Language differs dramatically from context to context. To some degree, large language models like GPT-3 account for such variation by conditioning on strings of initial input text, or prompts. However, prompting can be ineffective when contexts are sparse, out-of-sample, or extra-textual. In this paper, we introduce the mixed-effects transformer (MET), a novel approach for learning hierarchically-structured prefixes— lightweight modules prepended to an input sequence— to account for structured variation in language use. Specifically, we show how the popular class of mixed-effects regression models may be extended to transformer-based architectures using a regularized prefix-tuning procedure with dropout. We evaluate this approach on several domain-adaptation benchmarks, finding that it learns contextual variation from minimal data while generalizing well to unseen contexts.
BibTeX
@inproceedings{white2022mixed,
  author = {Julia White and Noah D. Goodman and Robert X. D. Hawkins},
  title = {{Mixed-effects transformers for hierarchical adaptation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=bEmrBheiq7},
}
Figure