Introduction
While deep learning models are typically compressed to optimize for storage and inference speed, recent research suggests a surprising secondary benefit: compression can actually improve a model's ability to understand human disagreement. In Natural Language Understanding (NLU) tasks like sentiment analysis, humans often disagree on labels—a phenomenon known as label ambiguity. Standard deep models, such as BERT or RoBERTa, tend to become "over-confident" as they get deeper, producing "peaky" probability distributions that ignore the nuances of human opinion. This paper demonstrates that by pruning these over-confident top layers and distilling "relationship knowledge" from lower layers, models can become both more efficient and better at capturing ambiguity.
The Problem of Over-Confidence
Current state-of-the-art models are usually trained on "hard" labels (single ground-truth answers). As these models increase in depth, they learn to separate classes so distinctly that they lose the underlying relationships between similar categories. For instance, a model might become so certain a sample is "Fear" that it completely ignores its similarity to "Sadness," even if human annotators are split between the two. While techniques like Label Smoothing or Monte Carlo (MC) Dropout attempt to address this, they often lack the ability to capture specific class relationships or add significant computational cost.
Methodology: Pruning and Distillation
The authors propose a two-step strategy based on the observation that middle-to-lower layers of a Transformer represent class relationships more accurately than the final layers.
- Layer Pruning: The researchers identify a "target layer"—the lowest layer that maintains validation accuracy within 1% of the full model—and prune all subsequent layers. This effectively removes the layers where the model typically over-fits into an over-confident state.
- Relationship Knowledge Distillation (KD): To refine the pruned model, they use an internal distillation process. Instead of an external teacher, they treat a lower, "ambiguity-rich" layer as the source. The final classifier of the pruned network is trained to match the distribution of this lower layer while still adhering to the gold labels.
Visualizing the "Over-Confidence Filter"
The core insight of this work is that deeper layers act as a filter that strips away ambiguity. Visualization using t-SNE confirms that as data passes through the final layers, class clusters become extremely distinct and distant.
As shown in the figure above, the 12th layer has the lowest entropy (0.1911), indicating high confidence but poor representation of class overlap. In contrast, the 11th layer maintains the highest validation accuracy (87.61%) while preserving more distributional information (entropy of 0.3246).
Key Results and Efficiency
The proposed method was evaluated on datasets with high human disagreement, such as ChaosSNLI, ChaosMNLI, and SemEval-2007 Task 14.
- Improved Ambiguity Capture: On the ChaosMNLI dataset, the method achieved a Jensen-Shannon Divergence (JSD) of 0.2799, a significant improvement over the standard model's 0.4219.
- Model Compression: The researchers successfully pruned 3 layers from standard NLU models, reducing parameter counts from 125M to 103M.
- Latency Reduction: On mobile hardware (Samsung Galaxy Tab S6 Lite), inference latency dropped from 2.42 seconds to 1.86 seconds per 300 tokens.
- Synergistic Effects: The method was found to be complementary to Label Smoothing, further enhancing the ability of already-regularized models to capture human opinion distributions.
Conclusion
This research highlights that model compression is not just a trade-off between size and performance. By strategically pruning over-confident layers and distilling internal relationship knowledge, practitioners can create NLU models that are faster, smaller, and more aligned with the inherent ambiguity of human language. Future work will look into more granular compression techniques, such as pruning specific attention heads, to see if they offer similar benefits for capturing nuance.
Summary updated