PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression

Lizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, Shiguang NiarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) are deployed for increasingly complex tasks, the length of input prompts has grown significantly. This growth introduces a critical computational bottleneck: memory consumption scales quadratically with sequence length, while latency increases linearly. Current solutions to this problem typically involve slow generative models for rewriting or heuristic pruning methods that often discard vital information by ignoring the model's internal logic.

To address these challenges, researchers have introduced Prompt Importance Sampling (PIS). This framework slashes prompt length by formally linking LLM attention scores to the statistical theory of importance sampling. By treating prompt compression as a sampling problem, PIS reduces inference overhead by up to 38% while maintaining, or even improving, reasoning accuracy.

The PIS Framework

The PIS framework operates through a dual-level architecture designed to identify and preserve the most informative elements of a prompt while discarding redundant or distracting "noise."

1. Token-Level Importance Sampling (TIS)

At the granular level, PIS employs a lightweight, 9-layer Double Deep Q-Network (DDQN) to calculate saliency scores for individual tokens. This RL agent combines attention scores from a small encoder (such as BERT) with TF-IDF metrics. This ensures that tokens which are both heavily attended to by the model and statistically informative within the text are prioritized. The agent adaptively determines compression ratios by analyzing the context of surrounding sentences, allowing it to preserve complex logical instructions while heavily compressing simpler phrases.

2. Sentence-Level Importance Sampling (SIS)

To manage redundancy across longer contexts, PIS implements a Russian Roulette strategy. When a new sentence exhibits high cosine similarity (≥ 0.9) to a previous one, the model assigns it a deletion probability. This probability scales with the number of redundant sentences encountered, effectively pruning the repetitive "fluff" often found in meeting transcripts or long-form documents.

The PIS framework: from sentence segmentation to dual-level importance sampling.

Key Insights: Attention as Importance

A core theoretical contribution of this work is the mathematical link established between a Transformer's attention scores and Importance Sampling weights. By reframing the prompt as a distribution to be sampled, the authors provide a rigorous foundation for token deletion. Their research indicates that tokens with high variance in attention scores are often the most critical for logical reasoning; prioritizing these tokens helps maintain the "logical skeleton" of the prompt even at high compression ratios.

Experimental Results

The authors evaluated PIS across diverse benchmarks, including MeetingBank (summarization and QA), GSM8K (math reasoning), and BBH (general logic).

Performance and Efficiency

PIS consistently outperformed strong baselines like LLMLingua and LLMLingua-2. On the MeetingBank dataset, PIS achieved an 89.05% Exact Match (EM) score for QA tasks, notably outperforming the "no compression" baseline of 88.14%. This suggests that PIS acts as a "denoiser," removing irrelevant tokens that might otherwise distract the LLM during inference.

Performance comparison on MeetingBank QA and summarization tasks.

In terms of speed, PIS demonstrated significant latency improvements. For a 1,500-token input at 5x compression, PIS required only 2.65 seconds, compared to 3.85 seconds for LLMLingua-2. Furthermore, the method showed robust generalization, nearly matching full-shot performance on out-of-domain tasks (GSM8K and BBH) while using significantly fewer tokens.

Limitations and Future Work

While PIS represents a significant step forward, several areas for improvement remain:

  • Segmentation: The current reliance on punctuation-based splitting may not translate well to technical code or documents lacking standard punctuation.
  • Training Costs: Although the final model is lightweight, training the RL agent requires iterative LLM evaluations. Future iterations may utilize distilled reward models to reduce this initial overhead.
  • Flexibility: The authors aim to develop more flexible architectures that can handle arbitrary compression targets without requiring specific retraining for different ratios.

Summary updated

BibTeX
@misc{chen2025pis,
  author = {Lizhe Chen and Binjia Zhou and Yuyao Ge and Jiayi Chen and Shiguang NI},
  title = {{PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression}},
  year = {2025},
  eprint = {2504.16574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.16574},
}
Figure