Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking

Guojie Liu, Yiqi Wang, Yanfeng Yang, Wenqi Fan, Songlei Jian, Jianfeng Zhang, Jie YuarXiv 2026 · arXiv

TL;DR

Soft prompt compression condenses long LLM contexts into compact "memory embeddings" to reduce inference latency, but standard approaches are slow to train due to unconstrained global attention. This paper introduces Parallelized Iterative Compression (PIC), which uses a block-wise causal mask to mimic human memory chunking. This simple architectural tweak accelerates compressor training by 40% and significantly boosts downstream performance, particularly at high compression ratios (up to 64×).

What Problem Does This Paper Address?

As Large Language Models (LLMs) handle increasingly massive contexts for tasks like Retrieval-Augmented Generation (RAG) and In-Context Learning (ICL), they hit a computational bottleneck: the cost of self-attention scales quadratically with sequence length, causing severe inference latency.

"Soft prompt compression" mitigates this by training a small compressor model to distill long context blocks into a few dense, continuous embedding vectors (memory tokens). However, existing methods allow every memory token to attend to the entire context indiscriminately. Learning these complex global routing patterns from scratch makes compressor training incredibly data-hungry, unstable, and slow.

Methods

The authors observed that when standard global compressors finally converge, they naturally develop "spatial specialization"—meaning individual memory tokens implicitly learn to focus on specific, contiguous blocks of text.

To bypass the painful process of learning this configuration from scratch, PIC explicitly bakes this structural inductive bias directly into the Transformer's attention mechanism using a block-wise causal attention mask.

The process works as follows:

  1. The input context is divided into $N$ equal-length, non-overlapping chunks.
  2. $N$ learnable memory tokens are appended to the sequence, and the entire sequence is processed in parallel.
  3. A custom attention mask enforces three strict visibility rules:
  4. Intra-Context Causal: Context tokens follow standard causal masking.
  5. Intra-Memory Causal: Memory tokens can only attend to preceding memory tokens.
  6. Memory-to-Block Attention: The $t$-th memory token can only attend to its designated $t$-th context chunk and the historical memory tokens.

This design perfectly replicates a sequential, chunk-by-chunk human reading strategy while maintaining the computational efficiency of a single parallel forward pass.

Figure 4: Comparison of different compression paradigms. (a) Direct Compression: Each memory token attends globally to the entire input context without structural constraints. (b) Iterative Compression: Inspired by the chunking mechanism in human working memory, we split the input context into chunks, and memory tokens are generated sequentially, with each token attending only to its specific chunk and previous memory. (c) Parallelized Iterative Compression (PIC): Our proposed method allows all memory tokens to be processed in parallel within a single sequence, using a block-wise causal mask to enforce the same local dependency constraints as the iterative approach.

Datasets

  • Pre-training: ~3 billion tokens sampled from the FineWeb dataset.
  • Fine-tuning & Evaluation (RAG QA): SQuAD (primary training/adaptation set), HotpotQA, AdversarialQA, and Natural Questions (NQ).
  • Fine-tuning & Evaluation (In-Context Learning): GSM8K (primary adaptation set), SST-2, and the Winograd Schema Challenge (WSC).

Key Insights and Ideas

  • Bio-Inspired Inductive Bias: Human working memory naturally groups continuous streams of information into higher-order units ("chunks") to overcome capacity limits. Mirroring this cognitive mechanism simplifies the model's objective from global dependency modeling to localized information extraction.
  • Mitigating Mode Collapse: In standard global compression, memory tokens often suffer from high redundancy or semantic overlap. By restricting each token to a unique text block, PIC forces the model to generate more distinct, orthogonal, and information-dense representations.
  • Training Stability: Unconstrained global attention baselines often degrade in performance if trained past a certain point. PIC's localized constraint provides a regularizing effect that ensures stable, continuous performance gains as more pre-training data is introduced.
Figure 5: Comparison of Reconstruction Loss During Pre- training. The upper plot illustrates the overall loss trajectory, while the lower plot presents a zoomed-in view of steps 10k to 30k, highlighting the faster convergence of our PIC (red) compared to the PCC baseline (dashed gray).

Key Results

  • 40% Faster Training: When training a 16× compressor on two 48GB RTX 4090 GPUs, PIC surpassed the peak performance of the competitive baseline in just 56 hours, effectively slashing training time by roughly 40%.
  • Superior High-Ratio Compression: The performance gap widens dramatically under tight memory budgets. At a extreme 64× compression ratio on QA tasks, PIC achieved a 29.8% relative improvement in F1 score and a 40.7% relative improvement in Exact Match (EM) score over the baseline.
  • Outperforming Uncompressed Context: On the WSC commonsense reasoning dataset, the 16× compressed PIC context actually outperformed the uncompressed few-shot baseline (65.38% vs 50.00% accuracy), indicating highly efficient retention of latent reasoning patterns.

Limitations and Future Work

The paper does not deeply detail its limitations, but a clear architectural constraint is its reliance on uniform chunking (splitting the context into equal-length, non-overlapping blocks). This rigid division can slice directly through sentences or paragraphs, potentially disrupting local semantic coherence.

The authors note that future work should continue exploring the intrinsic properties of text embeddings and suggest that spatial specialization patterns will remain a critical indicator of compressor efficacy. Adjacent extensions could involve exploring dynamic or semantic-based chunking boundaries rather than fixed token lengths.

Summary updated

Abstract

Providing extensive context via prompting is vital for leveraging the capabilities of Large Language Models (LLMs). However, lengthy contexts significantly increase inference latency, as the computational cost of self-attention grows quadratically with sequence length. To mitigate this issue, context compression-particularly soft prompt compressio-has emerged as a widely studied solution, which converts long contexts into shorter memory embeddings via a trained compressor. Existing methods typically compress the entire context indiscriminately into a set of memory tokens, requiring the compressor to capture global dependencies and necessitating extensive pre-training data to learn effective patterns. Inspired by the chunking mechanism in human working memory and empirical observations of the spatial specialization of memory embeddings relative to original tokens, we propose Parallelized Iterative Compression (PIC). By simply modifying the Transformer's attention mask, PIC explicitly restricts the receptive field of memory tokens to sequential local chunks, thereby lowering the difficulty of compressor training. Experiments across multiple downstream tasks demonstrate that PIC consistently outperforms competitive baselines, with superiority being particularly pronounced in high compression scenarios (e.g., achieving relative improvements of 29.8\% in F1 score and 40.7\% in EM score on QA tasks at the $64\times$ compression ratio). Furthermore, PIC significantly expedites the training process. Specifically, when training the 16$\times$ compressor, it surpasses the peak performance of the competitive baseline while effectively reducing the training time by approximately 40\%.
BibTeX
@misc{liu2026cognitive,
  author = {Guojie Liu and Yiqi Wang and Yanfeng Yang and Wenqi Fan and Songlei Jian and Jianfeng Zhang and Jie Yu},
  title = {{Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking}},
  year = {2026},
  eprint = {2602.13980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.13980},
}
Figure