Abstract
Large-scale language models are computationally intensive and often struggle with domain shift—performance degradation when moving from training data to real-world, unseen distributions. We introduce AMoC (ATE-guided Model Compression), a novel framework that leverages causal inference to prune models for better domain generalization. By estimating the Average Treatment Effect (ATE) of specific layers on model predictions using unlabeled target data, AMoC identifies essential components for cross-domain performance. Our results show that AMoC-compressed models not only outperform standard compression baselines but can even exceed the performance of the original uncompressed teacher model in out-of-distribution scenarios.
Introduction
The proliferation of massive Natural Language Processing (NLP) models has led to significant computational overhead. While techniques like DistilBERT offer a path toward efficiency, they typically focus on mimicking a teacher model within the same data distribution. In practice, models frequently encounter domain shift (e.g., transitioning from book reviews to electronics reviews), where standard compression methods often fail because they do not distinguish between layers that capture generalizable features and those that memorize domain-specific noise or "spurious correlations."
Methodology: The AMoC Framework
The core of AMoC is the application of causal inference to model architecture. By treating individual layers as "treatments," we can quantify their impact on the final output.
Causal Effect Estimation
We define the Average Treatment Effect (ATE) as the divergence in the model’s output distribution when a specific layer is present versus when it is bypassed. A key advantage of this metric is that it can be calculated using unlabeled data from the target domain, making it highly practical for real-world adaptation where labeled data is scarce.
The AMoC Algorithm
The compression process follows three main stages:
- Candidate Generation: Multiple compressed architectures are generated by removing various subsets of layers.
- Efficient Fine-tuning: To avoid the prohibitive cost of full retraining, AMoC freezes the majority of parameters and only fine-tunes the "bridge" connections between the remaining layers and the decoder.
- Stepwise Regression for Selection: A regression model is trained to predict target-domain performance based on features like ATE (on source and target domains), in-domain accuracy, and domain similarity. This allows the system to select the optimal architecture for a new domain pair without requiring target labels.
Experimental Evaluation
We evaluated AMoC across three core NLP tasks using dozens of diverse domain pairs:
- Sentiment Classification: Amazon product reviews (6 domains).
- Natural Language Inference (NLI): MultiNLI corpus (6 domains).
- Named Entity Recognition (NER): OntoNotes 5.0 (6 domains).
Key Results and Insights
- Superior Generalization: AMoC consistently outperformed strong baselines such as DistilBERT, LayerDrop, and ALBERT. On the Amazon Reviews dataset, it improved average target-domain performance by up to 4.56% over the next best adaptation-aware baseline.
- Pruning as Regularization: In approximately 25% of experiments, the compressed AMoC model actually outperformed the original, uncompressed BERT-base model on the target domain. This suggests that causal pruning effectively removes domain-specific noise that leads to overfitting.
- Computational Efficiency: AMoC achieves an 11x speedup in training compared to the original BERT and is 6x faster than DistilBERT, primarily due to its selective fine-tuning approach.
- Layer Importance: Analysis revealed that middle layers (e.g., layer 4 in BERT-base) are often the most critical for performance, whereas the final layers (11 and 12) are frequently redundant for domain generalization.
Conclusion and Future Work
AMoC demonstrates that causal inference provides a principled and efficient metric for model compression in domain adaptation contexts. By focusing on the effect of components rather than just weight magnitudes, we can create smaller, faster models that generalize better. Future research will explore applying this causal logic to finer-grained components, such as individual attention heads, and scaling the framework to "extra-large" models like T5 or GPT-series architectures.
Summary updated