More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss

Runsong Zhao, Pengcheng Huang, Xinyu Liu, Chunyang Xiao, Tong Xiao, Jingbo Zhu

Abstract

Compressing Transformer inputs into compressed tokens allows running LLMs with improved speed and cost efficiency. Based on the compression method ICAE, we carefully examine the position identifier choices for compressed tokens and also propose a new compression loss. We demonstrate empirically that our proposed methods achieve significantly higher compression ratios (15x compared to 4x for ICAE), while being able to attain comparable reconstruction performance.

Introduction

As Large Language Models (LLMs) are tasked with processing increasingly massive amounts of data, context compression has emerged as a vital field for maintaining efficiency. A popular technique involves using "special tokens" (soft prompts) that act as a summary, condensing long texts into a few dense vectors. However, the performance of these compressed tokens is heavily influenced by how they are represented within the model's positional framework.

The Problem: Positional Distance in Compression

Most existing compression frameworks, such as ICAE, utilize a Default Position Layout (DPL). In this setup, special memory tokens are assigned position IDs that place them at the very end of the sequence, following the entire context.

This approach suffers from a critical flaw: Transformer architectures possess a "local inductive bias." Due to the mechanics of position encodings like RoPE (Rotary Positional Embeddings) or Alibi, tokens naturally "prefer" to attend to their immediate neighbors in the position ID space. In a DPL setup, a special token intended to compress the beginning of a document might be assigned a position ID thousands of units away from its target text. This positional distance makes it significantly harder for the model to learn holistic dependencies, leading to inefficient compression and slower training convergence.

Proposed Method: Enhanced Position Layout (EPL)

To address these limitations, we introduce the Enhanced Position Layout (EPL). This method is "plug-and-play," requiring no architectural changes, only a modification of the numerical identifiers passed to the embedding layer. EPL consists of two primary components:

1. Uniform Position Layout (UPL)

Instead of bunching special tokens at the end of a sequence, UPL redistributes them across the context. It calculates an optimal "spread" so that every context token is as close as possible to at least one special token in the position ID space. Mathematically, it minimizes the maximum distance between any context token and its nearest "compressor" token.

Comparison between Default Position Layout (DPL) and Uniform Position Layout (UPL).

2. Consistent Position Layout (CPL)

During the decoding phase—when the model utilizes compressed tokens to generate an answer—many frameworks inadvertently break the causal order of position IDs. CPL ensures that the logical sequence (context, followed by compressed tokens, then the query) maintains a strictly increasing and consistent set of position IDs, preserving the model's internal logic.

Position ID assignments during the decoding phase for DPL vs EPL.

Key Insights and Results

Our empirical evaluations across text and vision modalities reveal several key advantages of the EPL approach:

  • Training Efficiency: Using EPL allowed the model's autoencoding loss to converge 9.7 times faster than the default layout. This suggests that models previously "struggled" against the constraints of distant position IDs.
  • Attention Optimization: Visualization of attention maps confirms that under UPL, special tokens focus more effectively on their surrounding context. In contrast, DPL often results in tokens defaulting to self-attention or noisy patterns due to the context being too "far" away in the ID space.
  • Performance Gains: Integrating EPL led to a 1.9 ROUGE-1 F1 improvement on out-of-domain question-answering tasks and a 1.8 BLEU gain on reconstruction tasks.
  • Cross-Modality Success: The benefits extend to vision-language models. For vision compression LLMs, EPL resulted in an average accuracy increase of 2.6 points across multimodal benchmarks (SQA, MMB, and VQAv2).

Conclusion

By recognizing that position IDs are "virtual" and can be manipulated to minimize distance between context and compressors, we significantly improve LLM efficiency. EPL allows for higher compression ratios (up to 15x) while maintaining high reconstruction quality. Future work will explore dynamic layouts that adapt based on content importance to further push the boundaries of lossy compression.

Summary updated

Abstract

Compressing Transformer inputs into compressd tokens allows running LLMs with improved speed and cost efficiency. Based on the compression method ICAE, we carefully examine the position identifier choices for compressed tokens and also propose a new compression loss. We demonstrate empirically that our proposed methods achieve significantly higher compression ratios (15x compared to 4x for ICAE), while being able to attain comparable reconstruction performance.
BibTeX
@inproceedings{zhao2025more,
  author = {Runsong Zhao and Pengcheng Huang and Xinyu Liu and Chunyang Xiao and Tong Xiao and Jingbo Zhu},
  title = {{More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure