SirLLM: Streaming Infinite Retentive LLM

Yao Yao, Zuchao Li, Hai Zhao

Abstract

As Large Language Models (LLMs) become increasingly prevalent in various domains, their ability to process inputs of any length and maintain a degree of memory becomes essential. However, the one-off input of overly long texts is limited, as studies have shown that when input lengths exceed the LLMs’ pre-trained text length, there is a dramatic decline in text generation capabilities. Moreover, simply extending the length of pre-training texts is impractical due to the difficulty in obtaining long text data and the substantial memory consumption costs this would entail for LLMs. Recent efforts have employed streaming inputs to alleviate the pressure of excessively long text inputs, but this approach can significantly impair the model’s long-term memory capabilities. Motivated by this challenge, we introduce Streaming Infinite Retentive LLM (SirLLM), which allows LLMs to maintain longer memory during infinite-length dialogues without the need for fine-tuning. SirLLM utilizes the Token Entropy metric and a memory decay mechanism to filter key phrases, endowing LLMs with both long-lasting and flexible memory. Our experimental results robustly demonstrate that SirLLM can achieve stable and significant improvements across different LLMs and tasks, compellingly proving its effectiveness.

Introduction

Standard Large Language Models (LLMs) are typically constrained by a fixed context window. When inputs exceed this pre-trained length, performance collapses. Existing solutions like Sliding-window attention or StreamLLM (which retains initial "anchor" tokens and recent context) manage memory footprint but suffer from a "forgetting" problem. These methods prioritize recency over relevance, causing the model to lose track of critical information mentioned earlier in a long dialogue. SirLLM addresses this by selectively deciding what is worth remembering based on information density.

Methods

SirLLM transforms a standard LLM into a "Retentive" one through two primary mechanisms:

  1. Token Entropy Filtering: Instead of keeping only recent tokens, SirLLM calculates the "entropy" of each token ($e_i = -\log P(x_i | x_{<i})$). High entropy indicates a token was less predictable and carries more unique information (e.g., specific nouns or facts). SirLLM populates the Key-Value (KV) cache with these high-entropy tokens.
  2. Memory Decay Mechanism: To prevent the cache from becoming filled with outdated information, a decay ratio ($\eta_{decay} < 1$) is applied. After each dialogue turn, the entropy scores of stored tokens are multiplied by this ratio, allowing distant information to fade naturally.
Comparison of attention mechanisms: (a) Dense, (b) Sliding-window, (c) Attention Sink (StreamLLM), and (d) Token Entropy (SirLLM).

Experimental Evaluation

We evaluated SirLLM across three distinct tasks designed to test memory and pattern recognition:

  • DailyDialog: Tests conversational coherence in multi-turn dialogues.
  • Grocery Shopping: A recall benchmark where the model must remember a list of items after 20 rounds of distracting common-sense QA.
  • Rock-Paper-Scissors: Tests the model's ability to recognize and adapt to long-term opponent patterns over 2,000 rounds.
  • Wikitext-103: Measures language modeling perplexity (PPL) over 20,000-token sequences.

Key Results and Insights

SirLLM demonstrates that entropy is a strong proxy for importance. LLMs naturally assign higher attention weights to tokens that are harder to predict. By leveraging this internal signal, SirLLM achieves superior recall. In the Grocery Shopping task, SirLLM achieved nearly 100% accuracy, while StreamLLM dropped to ~25%.

Attention weights in Layer 32, demonstrating the concentration of attention on the initial token (sink) and the local diagonal context.

Furthermore, SirLLM maintained stable perplexity on long sequences, matching StreamLLM's efficiency while providing significantly better long-term memory. In the Rock-Paper-Scissors test, SirLLM was the only model to consistently improve its win rate by "remembering" historical preferences, proving its ability to handle infinite context without fine-tuning.

Conclusion

SirLLM provides a zero-shot solution for infinite context by combining token entropy filtering with a flexible memory decay mechanism. While future work may explore adaptive decay ratios, SirLLM currently stands as a robust framework for enabling long-term memory in streaming LLM applications.

Summary updated

Abstract

As Large Language Models (LLMs) become increasingly prevalent in various domains, their ability to process inputs of any length and maintain a degree of memory becomes essential. However, the one-off input of overly long texts is limited, as studies have shown that when input lengths exceed the LLMs’ pre-trained text length, there is a dramatic decline in text generation capabilities. Moreover, simply extending the length of pre-training texts is impractical due to the difficulty in obtaining long text data and the substantial memory consumption costs this would entail for LLMs. Recent efforts have employed streaming inputs to alleviate the pressure of excessively long text inputs, but this approach can significantly impair the model’s long-term memory capabilities.Motivated by this challenge, we introduce Streaming Infinite Retentive LLM (SirLLM), which allows LLMs to maintain longer memory during infinite-length dialogues without the need for fine-tuning. SirLLM utilizes the Token Entropy metric and a memory decay mechanism to filter key phrases, endowing LLMs with both long-lasting and flexible memory. We designed three distinct tasks and constructed three datasets to measure the effectiveness of SirLLM from various angles: (1) DailyDialog; (2) Grocery Shopping; (3) Rock-Paper-Scissors. Our experimental results robustly demonstrate that SirLLM can achieve stable and significant improvements across different LLMs and tasks, compellingly proving its effectiveness. When having a coversation, “A sir could forget himself,” but SirLLM never does! Our code is publicly available at https://github.com/Zoeyyao27/SirLLMhttps://github.com/Zoeyyao27/SirLLM
BibTeX
@inproceedings{yao2024sirllm,
  author = {Yao Yao and Zuchao Li and Hai Zhao},
  title = {{SirLLM: Streaming Infinite Retentive LLM}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=agOWb2kL0O},
}
Figure