The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains

Haoran Xu, Philipp Koehn, Kenton Murray

TL;DR

Instead of pruning "redundant" parameters that contribute little to a model, this paper proposes intra-distillation: a training method that forces all parameters to contribute equally. By balancing parameter sensitivity, the authors achieve significant performance gains across machine translation, NLU, and cross-lingual tasks without increasing model size.

What Problem Does This Paper Address?

Modern deep learning models are notoriously over-parameterized. Research into "lottery tickets" and model pruning has shown that we can often remove up to 80% of a model's weights with negligible performance loss. This suggests that a vast majority of parameters are "redundant"—they have low sensitivity, meaning zeroing them out barely affects the model's output.

The authors of this paper ask a provocative question: What if these parameters aren't inherently useless, but just poorly trained?

They identify a massive "sensitivity gap" during standard training, where a small subset of parameters (the top 20%) does almost all the heavy lifting, while the rest contribute nearly nothing. Rather than discarding this "bottom 80%," the authors aim to "rejuvenate" them, encouraging a more balanced contribution across the entire network to improve generalization.

Figure 2: The mean sensitivity along with the number of training updates of top 20% most sensitive param- eters and the rest of 80% parameters. Though the top 20% of parameters have significant variance in sensitiv- ity at various stages of training, the bottom 80% consis- tently have very little sensitivity.

Methods

The core proposal is Intra-Distillation. Unlike standard knowledge distillation, which transfers knowledge from a large teacher to a smaller student, intra-distillation happens within a single model during training.

  1. Multi-Pass Training: For every training step, the model performs $K$ forward passes (typically $K=3$).
  2. Random Disabling: In each pass, a different random subset of parameters is "disabled" using dropout.
  3. Consistency Loss: The training objective includes a new term that minimizes the difference between these $K$ outputs. This effectively forces the model to produce the same result regardless of which specific parameters are active, thereby requiring all parameter groups to be equally capable and sensitive.
Figure 1: Illustration of the intra-distillation process. We pass the model K (K = 3) times and obtain three outputs p1, p2, p3. Each time we disable different subsets of parameters (illustrated by different colors). Minimizing the difference of the K outputs approxi- mates minimizing the contribution difference of these disabled subsets of parameters, which can significantly improve model generalization performance.

To make this work effectively, the authors introduced two technical refinements:

  • X-divergence: A novel, bounded similarity metric for classification tasks that outperforms standard Jensen-Shannon divergence.
  • Adaptive Strength Control: A scheduling algorithm for the intra-distillation weight ($\alpha$). It starts small to allow initial convergence and ramps up in the middle of training to maximize the balancing effect.
Figure 4: An example of how α′ increase with the con- trol of p and q. The relationship between p

Datasets

The method was evaluated across a diverse set of NLP benchmarks:

  • Machine Translation: IWSLT’14 (8 language pairs, low-resource) and WMT’17 En$\rightarrow$De (high-resource).
  • Natural Language Understanding: The full GLUE benchmark (8 tasks including QNLI, MNLI, and SST-2).
  • Zero-Shot Cross-Lingual Transfer: XTREME-R benchmarks, specifically Named-Entity Recognition (NER) across 48 languages and TyDiQA (Question Answering) across 9 languages.

Key Insights and Ideas

  • Self-Distillation is Sensitivity Balancing: The authors provide a new explanation for why "self-distillation" (training a model to mimic an earlier version of itself) works. They show it naturally leads to a more concentrated, balanced distribution of parameter sensitivity.
  • Redundancy is a Training Failure: The success of pruning isn't necessarily a sign of efficient architecture, but rather a sign that standard training objectives fail to utilize the model's full capacity.
  • The "Brittle" Model Paradox: Interestingly, models trained with intra-distillation are much harder to prune. If you try to remove the "least important" 50% of weights from an intra-distilled model, performance collapses. This is actually a good sign—it proves that those previously "redundant" parameters are now doing vital work.
Comparison of sensitivity distributions: Intra-distillation (right) creates a much tighter, more balanced distribution than standard training (left).

Key Results

  • Machine Translation: On IWSLT'14, intra-distillation provided an average gain of 3.54 BLEU points over a standard Transformer. It even outperformed the "Switch Transformer," which has nearly double the parameters.
  • GLUE Benchmark: The method improved the average score of a BERT-base model by 0.81 points, showing particularly strong gains on larger datasets like QNLI (+1.1) and QQP (+1.3).
  • Cross-Lingual Transfer: In zero-shot settings, the method improved NER by 1.4 F1 and the more complex TyDiQA task by 2.2 F1 on average across all languages.
  • Efficiency: While training is slower, inference speed remains identical to the baseline because the final model architecture does not change.

Limitations and Future Work

  • Training Cost: The primary drawback is computational. Passing the model $K=3$ times per step roughly triples the training time (or memory requirement if processed in parallel).
  • Domain Scope: The study focused exclusively on NLP. While the method is task-agnostic, its effectiveness in computer vision or speech recognition remains to be explored.
  • Hyperparameter Tuning: The adaptive $\alpha$ schedule introduces new hyperparameters ($p, q$) that may require tuning for different architectures.

Future work could involve finding more efficient ways to simulate the $K$-pass consistency or applying these insights to design inherently more efficient, smaller architectures from the start.

Summary updated

Abstract

Recent model pruning methods have demonstrated the ability to remove redundant parameters without sacrificing model performance. Common methods remove redundant parameters according to the parameter sensitivity, a gradient-based measure reflecting the contribution of the parameters. In this paper, however, we argue that redundant parameters can be trained to make beneficial contributions. We first highlight the large sensitivity (contribution) gap among high-sensitivity and low-sensitivity parameters and show that the model generalization performance can be significantly improved after balancing the contribution of all parameters. Our goal is to balance the sensitivity of all parameters and encourage all of them to contribute equally. We propose a general task-agnostic method, namely intra-distillation, appended to the regular training loss to balance parameter sensitivity. Moreover, we also design a novel adaptive learning method to control the strength of intra-distillation loss for faster convergence. Our experiments show the strong effectiveness of our methods on machine translation, natural language understanding, and zero-shot cross-lingual transfer across up to 48 languages, e.g., a gain of 3.54 BLEU on average across 8 language pairs from the IWSLT'14 translation dataset.
BibTeX
@inproceedings{xu2022importance,
  author = {Haoran Xu and Philipp Koehn and Kenton Murray},
  title = {{The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.13},
}
Figure