Abstract
Large Language Models (LLMs) face significant latency challenges when processing long contexts, particularly on edge devices. While prompt compression offers a solution, existing BERT-based compressors suffer from "over-smoothing" and the loss of rare, high-information tokens. We introduce MOOSComp, a lightweight compressor that mitigates these issues through an Inter-Class Cosine Similarity Loss during training and an Outlier Score mechanism during compression. MOOSComp achieves a 3.3x speedup on mobile devices while maintaining superior accuracy across diverse benchmarks.
Introduction
The increasing context window of LLMs enables complex tasks but introduces substantial computational overhead. Hard prompt compression, which uses a smaller model to filter tokens, is a common strategy. However, state-of-the-art compressors like LLMLingua-2 often fail due to representational collapse—where "preserve" and "discard" tokens become increasingly similar in deeper layers—and the accidental removal of task-critical rare entities like names or dates.
Methodology
MOOSComp enhances the standard BERT-based compression pipeline with two key innovations designed to improve token classification and information retention.
1. Inter-Class Cosine Similarity Loss
To address over-smoothing, we introduce a novel loss term during the training phase. Standard cross-entropy loss focuses on classification accuracy but does not explicitly separate the underlying vector representations. Our similarity loss explicitly penalizes the model when the embeddings of "preserve" and "discard" tokens are too close in the vector space, forcing the encoder to maintain distinct boundaries between the two classes.
2. Outlier Score Integration
During the compression phase, we calculate an "outlier score" for each token based on its deviation from the class average. High-outlier tokens often represent rare entities or specific information crucial for task performance. By combining this score with the classifier's output, MOOSComp "rescues" important tokens that might otherwise be overlooked by a task-agnostic classifier.
Experimental Results
We evaluated MOOSComp on LongBench, GSM8K, and BBH datasets using a BERT-base encoder.
- Accuracy: MOOSComp consistently outperforms LLMLingua-2, especially at high compression ratios (e.g., 5x).
- Efficiency: On a V100 GPU, it provides a 3.2x speedup. On a mobile NPU, it achieves a 3.3x speedup at a 4x compression ratio.
- Qualitative Analysis: MOOSComp successfully preserves specific entities like "Alamo Bowl" or "Art Briles," which are often lost by baseline methods, directly improving the downstream LLM's reasoning capabilities.
Discussion and Conclusion
While highly effective for natural language, the outlier mechanism shows limitations in structured data like code, where "outliers" are less correlated with semantic importance. Future work will explore adaptive weighting for different domains. Overall, MOOSComp demonstrates that addressing representational collapse and preserving rare information are critical for efficient, high-performance long-context inference on the edge.
Summary updated