PSC: Extending Context Window of Large Language Models via Phase Shift Calibration

Wenqiao Zhu, Chao Xu, Lulu Wang, Jun WuEMNLP 2024 · DOI

Introduction

Modern Large Language Models (LLMs) predominantly utilize Rotary Position Embedding (RoPE) to encode token order. Extending the context window of these models (e.g., from 4k to 64k tokens) typically involves rescaling RoPE base frequencies. However, finding optimal scaling factors is an exponential search problem. Suboptimal frequencies introduce "full-rank" errors in the attention mechanism. Because standard fine-tuning methods like LoRA are "low-rank," they are mathematically ill-equipped to correct these discrepancies, leading to increased perplexity and performance degradation in long contexts.

Phase Shift Calibration (PSC)

To address this, the authors propose Phase Shift Calibration (PSC), a parameter-efficient adapter module that "fixes" suboptimal frequency scaling. PSC acts as a pre-processor, calibrating embeddings before they undergo positional encoding.

The PSC architecture integrates a small MLP to calibrate query and key embeddings before RoPE application.

The module consists of a two-layer Multilayer Perceptron (MLP) utilizing block diagonal matrices. This structure mirrors RoPE's operation on dimension pairs, allowing the model to learn transformations that align input embeddings with imperfect frequencies. By applying this calibration before RoPE (pre-calibration), the module avoids the non-linear distortions introduced by rotation, making the learning task more straightforward. For a LLaMA-2 7B model, PSC adds only ~64M parameters—less than 0.1% of the total model size.

Experimental Framework

The effectiveness of PSC was evaluated using a comprehensive suite of benchmarks:

  • Training: Models were fine-tuned on subsets of RedPajama and PG19 (chunked to 64k segments).
  • Evaluation: Perplexity was measured on PG19 and Proof-pile (specifically documents > 128k tokens).
  • Retrieval: The Passkey Retrieval task tested "needle-in-a-haystack" capabilities up to 36k tokens.
  • General Benchmarks: L-Eval for long-context tasks and standard benchmarks (MMLU, ARC) to ensure no regression in general capabilities.

Key Results and Insights

The core innovation of PSC lies in resolving the "Rank Inconsistency" between full-rank frequency errors and low-rank LoRA updates. By providing the necessary rank to bridge this gap, PSC significantly enhances performance.

PSC-enabled models maintain 100% retrieval accuracy significantly longer than standard fine-tuned variants.
  • Retrieval Accuracy: While standard fine-tuned models (e.g., YaRN FT) see accuracy drops as they approach their context limits, PSC-enabled models maintain 100% accuracy up to 34k tokens.
  • Perplexity Reduction: PSC consistently lowered perplexity across 16k, 32k, and 64k context windows. The performance gap between PSC-enabled models and standard models widened as context length increased, demonstrating superior scalability.
  • Real-World Tasks: On the L-Eval benchmark, LLaMA-2 7B with PSC improved its average score from 19.54 to 24.32, indicating better performance in summarization and long-form Q&A.
  • Efficiency: The computational overhead is minimal, adding only about 5ms of latency per 16k tokens and negligible GPU memory increases.

Conclusion

PSC offers a robust, parameter-efficient solution for extending LLM context windows. By shifting the focus from searching for "perfect" frequencies to calibrating embeddings for "good enough" frequencies, it provides a universal enhancement for RoPE-based models. While it requires a fine-tuning step, the resulting gains in retrieval accuracy and perplexity make it a compelling choice for long-context applications.

Summary updated

BibTeX
@inproceedings{zhu2024psc,
  author = {Wenqiao Zhu and Chao Xu and Lulu Wang and Jun Wu},
  title = {{PSC: Extending Context Window of Large Language Models via Phase Shift Calibration}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.341},
}
Figure