A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

Yan Li, Tianyi Zhang, Zechuan Li, Soyeon Caren Han

Introduction

Large Language Models (LLMs) are typically constrained by a fixed context window defined during pretraining. Extending this window often requires expensive fine-tuning or results in performance degradation due to Positional Out-of-Distribution (O.O.D.) issues. Greedy Attention Logit Interpolation (GALI) is a training-free method designed to overcome these limitations, enabling LLMs to process sequences significantly longer than their original training context without manual tuning or performance collapse.

Problem Statement: The Context Window Constraint

The primary challenge in length extrapolation is the numerical instability of positional coordinates, such as Rotary Position Embeddings (RoPE), when tokens exceed the training limit. Existing training-free solutions often rely on "global scaling," which stretches the coordinate system. While this accommodates longer sequences, it frequently introduces "logit outliers"—extreme values in the attention mechanism that drown out useful information and degrade the model's ability to retrieve relevant context, leading to a spike in perplexity.

Methodology

GALI addresses these challenges through two primary innovations: localized interpolation and logit-level adjustment.

1. Greedy and Localized Position ID Interpolation

Unlike global scaling methods that modify the entire sequence's coordinates, GALI preserves the original positional IDs for tokens within the training window. For tokens beyond this limit, it applies interpolation at a fine-grained, chunk-specific level. This "greedy" approach ensures that the model maintains its native performance on shorter contexts while providing a stable framework for extended sequences.

2. Logit-Level Interpolation with Positional Noise

Instead of interpolating high-dimensional embeddings, GALI operates directly on the raw attention logits before the softmax layer. It performs linear interpolation between known valid attention scores. To account for the oscillatory nature of RoPE—which linear interpolation tends to smooth out—GALI injects distance-scaled Gaussian noise. This noise acts as a regularizer, preventing the formation of logit outliers and maintaining numerical stability.

Overview of the GALI mechanism, illustrating Position ID Interpolation and Attention Logit Interpolation with Gaussian noise.

Experimental Evaluation

The effectiveness of GALI was evaluated using rigorous benchmarks, including LongBench (covering 16 tasks such as QA, summarization, and coding), L-Eval for closed-ended tasks, and PG19 for long-range language modeling. The experiments utilized backbone models such as Llama3-8B-instruct (with 4k and 8k context windows) and Llama2-7B-chat.

Results and Discussion

GALI demonstrates superior performance and stability across all tested benchmarks. On the LongBench suite, GALI-16k (using a Llama3-4k backbone) achieved an average score of 46.22, significantly outperforming other training-free methods like SelfExtend and NTK-Aware RoPE.

Comprehensive performance comparison of GALI against baseline and state-of-the-art training-free methods across various LongBench tasks.

Key findings include:

  • The "Narrower is Better" Insight: Mapping long-context information into narrower, familiar positional ranges yields better results than global stretching.
  • Stability at Scale: In PG19 tests up to 32k tokens, GALI maintained a stable perplexity of 11.05, whereas competing methods like NTK saw perplexity explode beyond 300.
  • Zero-Tuning Adaptability: GALI does not require input-length-specific hyperparameter tuning, making it highly practical for variable-length real-world documents.

Conclusion and Future Work

GALI provides a robust, training-free path for extending the context window of LLMs. By combining localized position interpolation with logit-level noise injection, it achieves state-of-the-art results without the need for retraining. While current implementations face computational overhead due to the dual-pass attention logit calculation—limiting compatibility with standard Flash Attention kernels—future work aims to integrate GALI's logic directly into optimized kernels to enhance speed and memory efficiency.

Summary updated

Abstract

Transformer-based Large Language Models (LLMs) struggle to process inputs exceeding their training context window, with performance degrading due to positional out-of-distribution (O.O.D.) that disrupt attention computations. Existing solutions, fine-tuning and training-free methods, are limited by computational inefficiency, attention logit outliers or loss of local positional information. To address this, we propose Greedy Attention Logit Interpolation (GALI), a training-free length extrapolation method that maximizes the utilization of pretrained positional intervals while avoiding attention logit outliers through attention logit interpolation. The result demonstrates that GALI consistently outperforms state-of-the-art training-free methods. Our findings reveal that LLMs interpret positional intervals unevenly within their training context window, suggesting that extrapolating within a smaller positional interval range yields superior results-even for short-context tasks. GALI represents a significant step toward resolving the positional O.O.D. challenge, enabling more reliable long-text understanding in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at https://github.com/AcademyCityL/GALI.
BibTeX
@misc{li2025training,
  author = {Yan Li and Tianyi Zhang and Zechuan Li and Soyeon Caren Han},
  title = {{A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)}},
  year = {2025},
  eprint = {2502.02659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.02659},
}
Figure