TL;DR
Instead of pruning "redundant" parameters that contribute little to a model, this paper proposes intra-distillation: a training method that forces all parameters to contribute equally. By balancing parameter sensitivity, the authors achieve significant performance gains across machine translation, NLU, and cross-lingual tasks without increasing model size.
What Problem Does This Paper Address?
Modern deep learning models are notoriously over-parameterized. Research into "lottery tickets" and model pruning has shown that we can often remove up to 80% of a model's weights with negligible performance loss. This suggests that a vast majority of parameters are "redundant"—they have low sensitivity, meaning zeroing them out barely affects the model's output.
The authors of this paper ask a provocative question: What if these parameters aren't inherently useless, but just poorly trained?
They identify a massive "sensitivity gap" during standard training, where a small subset of parameters (the top 20%) does almost all the heavy lifting, while the rest contribute nearly nothing. Rather than discarding this "bottom 80%," the authors aim to "rejuvenate" them, encouraging a more balanced contribution across the entire network to improve generalization.
Methods
The core proposal is Intra-Distillation. Unlike standard knowledge distillation, which transfers knowledge from a large teacher to a smaller student, intra-distillation happens within a single model during training.
- Multi-Pass Training: For every training step, the model performs $K$ forward passes (typically $K=3$).
- Random Disabling: In each pass, a different random subset of parameters is "disabled" using dropout.
- Consistency Loss: The training objective includes a new term that minimizes the difference between these $K$ outputs. This effectively forces the model to produce the same result regardless of which specific parameters are active, thereby requiring all parameter groups to be equally capable and sensitive.
To make this work effectively, the authors introduced two technical refinements:
- X-divergence: A novel, bounded similarity metric for classification tasks that outperforms standard Jensen-Shannon divergence.
- Adaptive Strength Control: A scheduling algorithm for the intra-distillation weight ($\alpha$). It starts small to allow initial convergence and ramps up in the middle of training to maximize the balancing effect.
Datasets
The method was evaluated across a diverse set of NLP benchmarks:
- Machine Translation: IWSLT’14 (8 language pairs, low-resource) and WMT’17 En$\rightarrow$De (high-resource).
- Natural Language Understanding: The full GLUE benchmark (8 tasks including QNLI, MNLI, and SST-2).
- Zero-Shot Cross-Lingual Transfer: XTREME-R benchmarks, specifically Named-Entity Recognition (NER) across 48 languages and TyDiQA (Question Answering) across 9 languages.
Key Insights and Ideas
- Self-Distillation is Sensitivity Balancing: The authors provide a new explanation for why "self-distillation" (training a model to mimic an earlier version of itself) works. They show it naturally leads to a more concentrated, balanced distribution of parameter sensitivity.
- Redundancy is a Training Failure: The success of pruning isn't necessarily a sign of efficient architecture, but rather a sign that standard training objectives fail to utilize the model's full capacity.
- The "Brittle" Model Paradox: Interestingly, models trained with intra-distillation are much harder to prune. If you try to remove the "least important" 50% of weights from an intra-distilled model, performance collapses. This is actually a good sign—it proves that those previously "redundant" parameters are now doing vital work.
Key Results
- Machine Translation: On IWSLT'14, intra-distillation provided an average gain of 3.54 BLEU points over a standard Transformer. It even outperformed the "Switch Transformer," which has nearly double the parameters.
- GLUE Benchmark: The method improved the average score of a BERT-base model by 0.81 points, showing particularly strong gains on larger datasets like QNLI (+1.1) and QQP (+1.3).
- Cross-Lingual Transfer: In zero-shot settings, the method improved NER by 1.4 F1 and the more complex TyDiQA task by 2.2 F1 on average across all languages.
- Efficiency: While training is slower, inference speed remains identical to the baseline because the final model architecture does not change.
Limitations and Future Work
- Training Cost: The primary drawback is computational. Passing the model $K=3$ times per step roughly triples the training time (or memory requirement if processed in parallel).
- Domain Scope: The study focused exclusively on NLP. While the method is task-agnostic, its effectiveness in computer vision or speech recognition remains to be explored.
- Hyperparameter Tuning: The adaptive $\alpha$ schedule introduces new hyperparameters ($p, q$) that may require tuning for different architectures.
Future work could involve finding more efficient ways to simulate the $K$-pass consistency or applying these insights to design inherently more efficient, smaller architectures from the start.
Summary updated