Abstract
As Large Language Models (LLMs) become increasingly prevalent in various domains, their ability to process inputs of any length and maintain a degree of memory becomes essential. However, the one-off input of overly long texts is limited, as studies have shown that when input lengths exceed the LLMs’ pre-trained text length, there is a dramatic decline in text generation capabilities. Moreover, simply extending the length of pre-training texts is impractical due to the difficulty in obtaining long text data and the substantial memory consumption costs this would entail for LLMs. Recent efforts have employed streaming inputs to alleviate the pressure of excessively long text inputs, but this approach can significantly impair the model’s long-term memory capabilities. Motivated by this challenge, we introduce Streaming Infinite Retentive LLM (SirLLM), which allows LLMs to maintain longer memory during infinite-length dialogues without the need for fine-tuning. SirLLM utilizes the Token Entropy metric and a memory decay mechanism to filter key phrases, endowing LLMs with both long-lasting and flexible memory. Our experimental results robustly demonstrate that SirLLM can achieve stable and significant improvements across different LLMs and tasks, compellingly proving its effectiveness.
Introduction
Standard Large Language Models (LLMs) are typically constrained by a fixed context window. When inputs exceed this pre-trained length, performance collapses. Existing solutions like Sliding-window attention or StreamLLM (which retains initial "anchor" tokens and recent context) manage memory footprint but suffer from a "forgetting" problem. These methods prioritize recency over relevance, causing the model to lose track of critical information mentioned earlier in a long dialogue. SirLLM addresses this by selectively deciding what is worth remembering based on information density.
Methods
SirLLM transforms a standard LLM into a "Retentive" one through two primary mechanisms:
- Token Entropy Filtering: Instead of keeping only recent tokens, SirLLM calculates the "entropy" of each token ($e_i = -\log P(x_i | x_{<i})$). High entropy indicates a token was less predictable and carries more unique information (e.g., specific nouns or facts). SirLLM populates the Key-Value (KV) cache with these high-entropy tokens.
- Memory Decay Mechanism: To prevent the cache from becoming filled with outdated information, a decay ratio ($\eta_{decay} < 1$) is applied. After each dialogue turn, the entropy scores of stored tokens are multiplied by this ratio, allowing distant information to fade naturally.
Experimental Evaluation
We evaluated SirLLM across three distinct tasks designed to test memory and pattern recognition:
- DailyDialog: Tests conversational coherence in multi-turn dialogues.
- Grocery Shopping: A recall benchmark where the model must remember a list of items after 20 rounds of distracting common-sense QA.
- Rock-Paper-Scissors: Tests the model's ability to recognize and adapt to long-term opponent patterns over 2,000 rounds.
- Wikitext-103: Measures language modeling perplexity (PPL) over 20,000-token sequences.
Key Results and Insights
SirLLM demonstrates that entropy is a strong proxy for importance. LLMs naturally assign higher attention weights to tokens that are harder to predict. By leveraging this internal signal, SirLLM achieves superior recall. In the Grocery Shopping task, SirLLM achieved nearly 100% accuracy, while StreamLLM dropped to ~25%.
Furthermore, SirLLM maintained stable perplexity on long sequences, matching StreamLLM's efficiency while providing significantly better long-term memory. In the Rock-Paper-Scissors test, SirLLM was the only model to consistently improve its win rate by "remembering" historical preferences, proving its ability to handle infinite context without fine-tuning.
Conclusion
SirLLM provides a zero-shot solution for infinite context by combining token entropy filtering with a flexible memory decay mechanism. While future work may explore adaptive decay ratios, SirLLM currently stands as a robust framework for enabling long-term memory in streaming LLM applications.
Summary updated