Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, Kentaro Inui

Abstract

Ensembling multiple models fine-tuned from a single pretrained checkpoint, such as BERT, often yields diminishing returns because the resulting models are highly correlated. This paper introduces Multi-Ticket Ensemble, a novel approach that leverages the Lottery Ticket Hypothesis to create diverse ensembles. By fine-tuning different sparse "winning ticket" subnetworks rather than full dense models, the authors demonstrate that intentional pruning can spark the predictive diversity needed to significantly outperform standard dense ensembles. The method is evaluated on the GLUE benchmark, showing that sparsity serves as a powerful mechanism for decorrelating model errors.

Introduction

In modern Natural Language Processing (NLP), the "pretrain-then-finetune" paradigm is the standard. However, the immense computational cost of pretraining massive models means most practitioners rely on a single publicly available checkpoint. To improve performance, it is common to fine-tune this single model multiple times with different random seeds to create an ensemble.

The fundamental limitation of this approach is a lack of diversity. Because these models originate from the exact same weight initialization and only differ by minor stochastic factors like dropout or data shuffling, their errors are highly correlated. An ensemble’s strength is derived from the unique "views" its members provide; if every member makes the same mistakes, the ensemble offers little benefit over a single model. This paper explores whether the Lottery Ticket Hypothesis (LTH)—the idea that dense networks contain sparse subnetworks capable of matching the full network's performance—can be used to break this correlation.

Methodology: Multi-Ticket Ensembles

The authors propose Multi-Ticket Ensemble (MTE), which generates ensemble members by identifying and training different sparse subnetworks. They explore three distinct strategies to find these "tickets":

  1. BASE-LT (Naive Pruning): This uses standard Iterative Magnitude Pruning (IMP). A model is fine-tuned, weights with the lowest magnitudes are pruned, and the remaining weights are reset to their original pretrained values. This process repeats until the desired sparsity is reached.
  2. ACTIVE-LT (Sequential Diversity): To force models to be structurally different, the authors apply an $L_1$ regularizer (magnitude decay) to weights that survived in previous ensemble members. This "pushes" the current model to rely on different parameters than its predecessors during the fine-tuning phase.
  3. RANDOM-LT (Parallel Diversity): A simpler, parallelizable version where each ensemble member is assigned a unique random $L_1$ regularization mask. This encourages each model to prune a different random subset of weights, ensuring structural diversity without the need for sequential training.

Experimental Setup

The researchers evaluated their approach using the GLUE (General Language Understanding Evaluation) benchmark, with a primary focus on tasks such as MRPC (Microsoft Research Paraphrase Corpus) and STS-B (Semantic Textual Similarity Benchmark). They also explored SST-2, MNLI, QNLI, and CoLA to identify the limitations of the approach. The experiments primarily utilized BERT-base-uncased, with additional validation performed using RoBERTa-base.

Key Insights and Results

Sparsity as a Source of Diversity

The most significant finding is that pruning is not merely a tool for model compression. By forcing a model to operate with only a fraction of its weights (e.g., 70% sparsity), the model is compelled to find a different "pathway" through the pretrained knowledge. This structural constraint leads to more diverse predictions.

Performance Gains

On the MRPC task, the RANDOM-LT ensemble achieved a gain of +1.52 points over its single-model counterpart. This significantly outperformed the standard dense ensemble baseline, which only saw a gain of +0.86. Interestingly, the parallelizable RANDOM-LT often performed as well as or better than the more complex, sequential ACTIVE-LT, suggesting that simple stochasticity in pruning masks is sufficient for high diversity.

Structural vs. Predictive Diversity

The authors observed that even when different random seeds led to nearly identical pruning masks (as seen in BASE-LT, where masks often overlapped by 99%), the resulting sparse models still produced more diverse predictions than dense models. However, intentionally increasing structural diversity through ACTIVE or RANDOM-LT provided the most substantial performance boosts.

The "Winning Ticket" Constraint

A critical requirement for MTE is that the subnetworks must be "winning tickets"—they must maintain high individual accuracy. On tasks like CoLA or QNLI, where the authors struggled to find sparse subnetworks that matched the performance of the dense model, the Multi-Ticket Ensemble failed to outperform the dense baseline. The ensemble gain cannot always compensate for a significant drop in the performance of individual members.

Conclusion and Future Work

Multi-Ticket Ensembles provide a promising path for improving model performance without the need for multiple pretrained checkpoints. By treating sparsity as a feature for diversity rather than just a method for efficiency, researchers can extract more value from existing pretrained architectures.

Future research directions include applying more sophisticated pruning algorithms (such as movement pruning) to find winning tickets on more challenging tasks and scaling this approach to even larger models like T5 or GPT-3, where the costs of redundant training are most prohibitive.

Summary updated

BibTeX
@inproceedings{kobayashi2022diverse,
  author = {Sosuke Kobayashi and Shun Kiyono and Jun Suzuki and Kentaro Inui},
  title = {{Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model}},
  booktitle = {BigScience @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=rCzgE3zHL-q},
}
Figure