Bidirectional Token Compression for Efficient LLM Inference

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 110 papers16 conceptsfrozen 2026-10-04

Large Language Models (LLMs) are increasingly bottlenecked by the "context tax." Whether it is a massive retrieval-augmented generation (RAG) prompt or a long-winded Chain-of-Thought (CoT) reasoning path, every token adds to the computational cost and latency. Traditional methods for shrinking these prompts often rely on "causal" metrics like perplexity—evaluating a token's importance based only on the words that came before it.

However, a new wave of research suggests that the most effective way to compress a prompt is to look at it from both ends. By using bidirectional encoders to evaluate token significance, researchers are achieving massive reductions in prompt size without losing the semantic "signal" required for the model to perform.

The Blind Spot of Causal Pruning

In a standard causal language model, the importance of a token is usually measured by how much it surprises the model given the preceding text. If a word is highly predictable, it is deemed redundant.

The flaw in this logic is that the "utility" of a token often depends on information that hasn't appeared yet. Consider a long document followed by a specific question at the very end. A causal model processing the document doesn't know what the question will be, so it cannot know which parts of the document are relevant. As Zhuoshi Pan and colleagues argue in LLMLingua-2, unidirectional models are fundamentally limited because they cannot account for this "future" context when deciding what to discard.

Seeing the Whole Sequence

To solve this, researchers are shifting from generative models to encoder-based architectures (like BERT or XLM-RoBERTa) for the task of compression. Because encoders use bidirectional self-attention, every token "sees" every other token in the prompt simultaneously.

In the LLMLingua-2 framework, prompt compression is treated as a token classification problem. Instead of predicting the next word, a small, fast encoder looks at the entire prompt and assigns a "keep" or "discard" label to every token. This allows the model to identify that a sentence at the beginning of a prompt is only redundant because of a clarifying statement that appears much later. This bidirectional view allows for 3x–6x faster compression speeds while maintaining higher fidelity than previous entropy-based methods.

Task-Awareness and Reinforcement

While seeing the whole prompt is a start, not all tokens are equally useful for every task. A token might be semantically "important" in a general sense but useless for answering a specific query.

Menglin Xia and Qianhui Wu addressed this in TACO-RL by introducing Reinforcement Learning into the compression pipeline. By using the REINFORCE algorithm, they fine-tune a bidirectional encoder to maximize the performance of the target LLM on specific tasks. This ensures the compressor learns to preserve the specific "bridge" tokens that connect early context to late-appearing queries, leading to performance boosts of up to 189% over task-agnostic methods.

Compressing the Reasoning Path

This bidirectional intuition extends beyond input prompts to the model's own internal "thoughts." In Chain-of-Thought (CoT) reasoning, models often generate "filler" tokens or redundant logical steps that don't contribute to the final answer.

Methods like BINGO and TokenSkip apply bidirectional scoring to these reasoning chains. As Lang Cao notes in the BINGO paper, a reasoning step's significance is often only clear in light of the final conclusion. By evaluating the significance of each step based on both the initial question and the eventual solution, A*-Thought can prune up to 50% of a reasoning chain. This "dual-perspective" approach ensures that the model doesn't just "fast-forward" through the logic, but specifically removes the steps that are irrelevant to the final output.

Beyond Text: Multimodal Redundancy

The need for bidirectional context is perhaps most acute in multimodal settings. In EchoingPixels, Chao Gong and team demonstrate that in audio-visual LLMs, an early visual frame might only become "important" when a specific sound occurs seconds later. Causal pruning would likely discard that frame before the audio cue ever arrives. By using a bidirectional encoder to score tokens across modalities, EchoingPixels can reduce the computational overhead of these models by 80–95% without losing the temporal relationships that define video and audio.

Summary of the Intuition

The shift toward bidirectional compression represents a move from local importance (how much information does this token add to the sequence so far?) to global utility (how much does this token contribute to the final goal?). By using small, efficient encoders to "pre-read" the entire context, these methods allow LLMs to focus their massive computational power only on the tokens that truly matter.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_token_compression_for_effi_26c0c47e,
  title = {Bidirectional Token Compression for Efficient LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-token-compression-for-efficient-llm-inference/rdr_26c0c47ef3bb47ef924a097354bf2993},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_token_compression_for_effi_26c0c47e_v1,
  title = {Bidirectional Token Compression for Efficient LLM Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-token-compression-for-efficient-llm-inference/rdr_26c0c47ef3bb47ef924a097354bf2993/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}