TL;DR
Large Language Models (LLMs) are often overparameterized for text embedding tasks. This paper demonstrates that pruning a significant portion of an LLM's layers (up to 30% with negligible performance loss, and up to 80% with a modest drop) can drastically reduce model size and inference time without sacrificing embedding quality. They also introduce L³Prune, a novel layer-pruning strategy based on initial loss, offering an efficient way to create smaller, performant text encoders.
What Problem Does This Paper Address?
Large Language Models (LLMs) have achieved remarkable success in various NLP tasks, but their massive size leads to high inference costs and memory requirements. While LLMs are increasingly being adapted for text embedding tasks (converting text to fixed-dimensional vectors), their large footprint remains a significant barrier. Traditional text embedding models, often based on BERT-like architectures, are typically smaller and more efficient. This paper addresses the challenge of making LLMs practical for text embedding by exploring aggressive layer pruning as a method to reduce their size and computational demands, aiming to bridge the gap between LLM capabilities and the efficiency needed for embedding applications.
Methods
The core of this paper's approach is layer pruning, specifically focusing on removing the last layers of a pre-trained decoder-only LLM. The authors propose a straightforward method: simply remove the last p% of layers from an LLM before fine-tuning it for text embedding tasks. This process is described as being easily implementable with just a few lines of code.
Beyond this simple layer dropping, they introduce L³Prune, a novel heuristic for identifying which layers to prune. L³Prune leverages the initial loss evolution across layers of an LLM. By analyzing how the loss changes layer by layer before any fine-tuning, they identify two minima: one before the midpoint of the layers and one after. These minima are hypothesized to represent optimal points for pruning, leading to two configurations: a "large" variant with minimal performance loss and a "small" variant for resource-constrained settings.
The pruned models are then fine-tuned using supervised contrastive training for a relatively short period (1000 steps). This training is presented as sufficient to "heal" the model and recover performance, negating the need for additional explicit fine-tuning steps often seen in prior work.
Datasets
The authors utilize a combination of datasets for their experiments:
- Training: They employ a subset of the Massive Text Embedding Benchmark (MTEB), specifically the MTEB-15 subset, which comprises approximately 1.5 million samples. This subset is designed to be representative of the full MTEB across various task categories.
- Evaluation: The primary evaluation is performed on the MTEB-15 subset itself.
- Baseline: For comparison, they also reference scores from established encoder-only models on the full MTEB.
The paper also lists numerous datasets used in prior work and for training the LLMs themselves, including ELI5, HotpotQA, FEVER, MIRACL, MS-MARCO, NQ, SNLI, MNLI, SQuAD, TriviaQA, Quora Duplicate Questions, Mr-TyDi, DuReader, and T2Ranking.
Key Insights and Ideas
- LLMs are Overparameterized for Text Embeddings: The most striking insight is that LLMs, despite their generative prowess, retain significant redundancy in their layers when adapted for text embedding. This suggests that their vast capacity is not fully utilized for this specific task.
- Last Layers are Often Expendable: The research strongly supports the idea that the later layers of LLMs can be pruned without substantial degradation in text embedding performance. This contrasts with some findings in generative tasks where later layers might be more critical.
- Initial Loss as a Pruning Oracle: The discovery that the initial layer-wise loss curve can effectively guide pruning is a key methodological contribution. This heuristic provides a principled way to identify optimal pruning points, reducing the need for extensive trial-and-error.
- Contrastive Training as "Healing": The finding that a short period of supervised contrastive training is sufficient to recover performance after pruning is significant. It implies that this training paradigm inherently adapts the remaining layers to compensate for the removed ones, making the process efficient.
- Parameter Count Matters More Than Layer Percentage: The results suggest that the final performance of a pruned model correlates more strongly with its total parameter count than with the exact percentage of layers removed. This provides a useful rule of thumb for resource allocation.
Key Results
- Significant Pruning Achieved: The paper demonstrates that up to 30% of layers can be pruned with negligible impact on performance. Even aggressive pruning of up to 80% yields reasonably effective text embedding models, with performance dropping from 64.9 to 59.8 on the highest-performing model.
- L³Prune Variants:
- The "large" variant of L³Prune prunes 69-89% of the original size (31-11% pruning) with a negligible performance drop (-0.2 on average, sometimes even an increase).
- The "small" variant prunes 16-36% of the original size (84-64% pruning) with a modest performance drop of -4.4 to -6.9.
- Efficiency Gains: Pruning leads to proportional reductions in memory and inference time. Training the pruned models is also highly efficient, taking less than an hour on an A100 GPU.
- Competitive with Encoder-Only Models: The pruned LLM-based models, especially the "large" variants, perform competitively with or better than established encoder-only models, despite the latter often requiring much more complex and resource-intensive training pipelines.
Limitations and Future Work
- Scope of Pruning: The work primarily focuses on pruning the last layers and on decoder-only LLMs. The authors acknowledge that this might not hold for extensive modifications to the architecture or for models significantly larger than 8 billion parameters.
- Generalizability: While the method is shown to be effective across several LLM families, model-specific experimentation might still be required due to variations in resilience to pruning.
- Supervised Setting: The current work focuses on the supervised fine-tuning setting. Future work could explore the effectiveness of this pruning strategy in unsupervised or zero-shot settings.
- Beyond Layer Pruning: While layer pruning is effective, future research could investigate other pruning techniques (e.g., structured pruning of attention heads or feed-forward networks) in conjunction with LLMs for text embeddings.
- Ethical Considerations: The authors note that making powerful embedding models more accessible could potentially lower the bar for malicious actors, though they also argue that embedding models generally have fewer avenues for malicious behavior compared to generative LLMs.
Summary updated