Introduction
Neural networks are notoriously over-parameterized. Research into the "Lottery Ticket Hypothesis" suggests that a vast majority of weights can be pruned with minimal performance loss. This phenomenon stems from a significant sensitivity gap: a small subset of parameters performs the bulk of the computation, while the remainder contribute negligibly.
This paper introduces Intra-Distillation, a training paradigm that challenges the notion of "redundant" parameters. Instead of discarding them, the authors propose forcing these "slackers" to work harder by balancing parameter sensitivity. This approach yields substantial generalization improvements across machine translation and natural language understanding tasks.
Methodology: Balancing the Load
The authors propose a task-agnostic objective designed to minimize the contribution gap between parameters through a process of internal consistency.
The core of the method involves:
- Multi-Pass Training: During each training step, the model executes $K$ forward passes (typically $K=3$).
- Dropout as a Proxy: Standard dropout is utilized in each pass to create distinct "sub-models." By randomly disabling neurons, each pass is forced to rely on a different subset of the total parameter pool.
- X-Divergence Loss: The X-divergence loss function measures the discrepancy between the $K$ outputs. Minimizing this divergence forces the model to ensure that any subset of parameters can produce the correct output, effectively "spreading the load" across the entire architecture.
- Adaptive Strength ($\alpha$): To prevent the additional loss from hindering early convergence, an adaptive schedule is employed where the intra-distillation weight $\alpha$ increases rapidly during the middle phase of training.
Key Insights: The Pruning Paradox
A compelling piece of evidence for the method's efficacy is the "Pruning Paradox." In standard models, performance remains stable even after significant pruning because many parameters are redundant. However, intra-distilled models experience a rapid performance decline when weights are removed.
This sensitivity to pruning is a mark of success: it demonstrates that parameters previously considered "redundant" have been successfully integrated into the model's functional core. The authors argue that redundancy is often a failure of the training process rather than an inherent property of the model size.
Experimental Results
The method was rigorously evaluated on several benchmarks, showing consistent improvements:
- Machine Translation: On the IWSLT’14 benchmark, the method achieved an average improvement of 3.54 BLEU points over a standard Transformer.
- Natural Language Understanding: It consistently outperformed existing consistency-regularization techniques like R-Drop and sensitivity-based methods like SAGE across the GLUE benchmark.
- Cross-Lingual Transfer: In zero-shot settings on the XTREME-R benchmark, the method improved F1 scores by 1.4 points on NER and 2.2 points on TyDiQA, suggesting more robust universal representations.
Conclusion
Intra-Distillation demonstrates that how parameters are utilized is as important as the total parameter count. While the method increases training computation due to multiple forward passes, it requires no additional parameters at inference time and provides a significant boost to model robustness and generalization. Future work may explore more sophisticated parameter-disabling mechanisms beyond simple dropout to further refine this balance.
Summary updated