The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains

Haoran Xu, Philipp Koehn, Kenton Murray

Introduction

Neural networks are notoriously over-parameterized. Research into the "Lottery Ticket Hypothesis" suggests that a vast majority of weights can be pruned with minimal performance loss. This phenomenon stems from a significant sensitivity gap: a small subset of parameters performs the bulk of the computation, while the remainder contribute negligibly.

This paper introduces Intra-Distillation, a training paradigm that challenges the notion of "redundant" parameters. Instead of discarding them, the authors propose forcing these "slackers" to work harder by balancing parameter sensitivity. This approach yields substantial generalization improvements across machine translation and natural language understanding tasks.

Methodology: Balancing the Load

The authors propose a task-agnostic objective designed to minimize the contribution gap between parameters through a process of internal consistency.

Figure 1: Intra-Distillation framework using multi-pass training to minimize output divergence between different parameter subsets.

The core of the method involves:

  1. Multi-Pass Training: During each training step, the model executes $K$ forward passes (typically $K=3$).
  2. Dropout as a Proxy: Standard dropout is utilized in each pass to create distinct "sub-models." By randomly disabling neurons, each pass is forced to rely on a different subset of the total parameter pool.
  3. X-Divergence Loss: The X-divergence loss function measures the discrepancy between the $K$ outputs. Minimizing this divergence forces the model to ensure that any subset of parameters can produce the correct output, effectively "spreading the load" across the entire architecture.
  4. Adaptive Strength ($\alpha$): To prevent the additional loss from hindering early convergence, an adaptive schedule is employed where the intra-distillation weight $\alpha$ increases rapidly during the middle phase of training.

Key Insights: The Pruning Paradox

A compelling piece of evidence for the method's efficacy is the "Pruning Paradox." In standard models, performance remains stable even after significant pruning because many parameters are redundant. However, intra-distilled models experience a rapid performance decline when weights are removed.

Figure 5: The Pruning Paradox—Intra-distilled models show a sharper performance drop when weights are removed, indicating that previously redundant parameters are now performing critical work.

This sensitivity to pruning is a mark of success: it demonstrates that parameters previously considered "redundant" have been successfully integrated into the model's functional core. The authors argue that redundancy is often a failure of the training process rather than an inherent property of the model size.

Experimental Results

The method was rigorously evaluated on several benchmarks, showing consistent improvements:

  • Machine Translation: On the IWSLT’14 benchmark, the method achieved an average improvement of 3.54 BLEU points over a standard Transformer.
  • Natural Language Understanding: It consistently outperformed existing consistency-regularization techniques like R-Drop and sensitivity-based methods like SAGE across the GLUE benchmark.
  • Cross-Lingual Transfer: In zero-shot settings on the XTREME-R benchmark, the method improved F1 scores by 1.4 points on NER and 2.2 points on TyDiQA, suggesting more robust universal representations.

Conclusion

Intra-Distillation demonstrates that how parameters are utilized is as important as the total parameter count. While the method increases training computation due to multiple forward passes, it requires no additional parameters at inference time and provides a significant boost to model robustness and generalization. Future work may explore more sophisticated parameter-disabling mechanisms beyond simple dropout to further refine this balance.

Summary updated

Abstract

Recent model pruning methods have demonstrated the ability to remove redundant parameters without sacrificing model performance. Common methods remove redundant parameters according to the parameter sensitivity, a gradient-based measure reflecting the contribution of the parameters. In this paper, however, we argue that redundant parameters can be trained to make beneficial contributions. We first highlight the large sensitivity (contribution) gap among high-sensitivity and low-sensitivity parameters and show that the model generalization performance can be significantly improved after balancing the contribution of all parameters. Our goal is to balance the sensitivity of all parameters and encourage all of them to contribute equally. We propose a general task-agnostic method, namely intra-distillation, appended to the regular training loss to balance parameter sensitivity. Moreover, we also design a novel adaptive learning method to control the strength of intra-distillation loss for faster convergence. Our experiments show the strong effectiveness of our methods on machine translation, natural language understanding, and zero-shot cross-lingual transfer across up to 48 languages, e.g., a gain of 3.54 BLEU on average across 8 language pairs from the IWSLT'14 translation dataset.
BibTeX
@inproceedings{xu2022importance,
  author = {Haoran Xu and Philipp Koehn and Kenton Murray},
  title = {{The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.13},
}
Figure