Schrodinger's Memory: Large Language Models

Wei Wang, Qing Li

Abstract

This paper investigates the theoretical nature of memory in Large Language Models (LLMs), proposing a framework based on the Universal Approximation Theorem (UAT). We argue that LLM memory exists in a "Schrödinger-like" indeterminate state, collapsing into specific information only upon querying. Through fine-tuning experiments on 4,000 poems, we demonstrate that Transformers act as dynamic function fitters, capable of near-perfect memorization while remaining sensitive to model scale, output complexity, and language density.

1. Introduction: The Black Box of Neural Memory

The mechanism by which Large Language Models (LLMs) store and retrieve information remains a central mystery in artificial intelligence. Unlike traditional computing systems that rely on explicit database storage (hard drives), LLMs exhibit memory through the configuration of their neural weights. This paper addresses the theoretical gap in defining internal model memory, moving beyond practical workarounds like context-window expansion or Retrieval-Augmented Generation (RAG) to explore the fundamental "recall" mechanism of neural networks.

2. Theoretical Framework: Dynamic Universal Approximation

The core of our proposal is the extension of the Universal Approximation Theorem (UAT) to the Transformer architecture. While standard UAT posits that a feedforward network can approximate any continuous function with fixed parameters, we argue that Transformers represent a "dynamic" implementation of this principle.

The Transformer architecture serves as a dynamic function approximator, where attention mechanisms modulate the function fitting process.

In this framework, the attention mechanism allows the model to dynamically adjust its internal functions based on input cues. "Remembering" is thus redefined as the process of the model dynamically fitting a function to produce a specific output sequence given a learned prompt. In this view, memory and reasoning are two sides of the same coin: both involve the model's ability to infer an outcome based on learned weights and current input.

3. Experimental Methodology

To validate this framework, we conducted memorization stress tests using models from the Qwen and Bloom families. The models were tasked with memorizing 2,000 Chinese poems (CN Poems dataset) and 2,000 English poems (ENG Poems dataset).

The training objective treated metadata—such as Dynasty, Author, and Title—as the input function domain, with the full poem text as the target output. This setup mirrors a student's rote memorization process, allowing us to measure the "storage" efficiency of the model's weights over 100 epochs of fine-tuning.

4. Results and Key Insights

Our experiments revealed that LLMs can achieve remarkable memorization accuracy through high-epoch fine-tuning, effectively "baking" the data into their parameters.

Qualitative examples of the Qwen1.5-0.5B-Chat model successfully recalling Chinese and English poetry after fine-tuning.

4.1 Quantitative Performance

The Qwen2-1.5B-Instruct model demonstrated the highest capacity, reaching a staggering 99.9% accuracy on English poetry and 96.9% on Chinese poetry. We observed a significant "length penalty": increasing the output requirement from 256 to 512 tokens caused accuracy to drop significantly (e.g., Qwen2-0.5B fell from 77.5% to 56.85%), highlighting the physical limits of weight-based storage density.

4.2 Language and Model Quality

Models generally found English poems easier to memorize than Chinese ones, likely due to the higher complexity and information density of Chinese characters. Furthermore, the quality of base pre-training mattered significantly; Qwen2-0.5B outperformed Qwen1.5-0.5B, suggesting that superior language comprehension facilitates more efficient memory retention.

5. Discussion: Schrödinger’s Memory and the Hippocampus

We introduce the concept of "Schrödinger’s Memory" to describe the state of information within an LLM. Information is not stored as discrete files but as a potentiality within the network parameters. It only "collapses" into a concrete observation (the response) when the model is queried. Without a query, the memory remains indeterminate and unobservable.

We also draw a functional parallel between LLM fine-tuning and the human hippocampus. In biological systems, the hippocampus acts as a switch for updating long-term memory. In our experiments, the fine-tuning process serves as this "switch," allowing the model to update its internal parameters to "fit" new information into its long-term weight distribution.

6. Conclusion and Future Directions

By framing memory as dynamic function approximation, we provide a theoretical bridge between training dynamics and emergent knowledge. However, limitations remain, particularly regarding the inability to "forget" or update specific outdated facts without retraining. Future research should focus on the "saturation point" of these models—determining the exact capacity limits before new memories begin to overwrite existing ones—and developing mechanisms for targeted memory management.

Summary updated

Abstract

Memory is the foundation of all human activities; without memory, it would be nearly impossible for people to perform any task in daily life. With the development of Large Language Models (LLMs), their language capabilities are becoming increasingly comparable to those of humans. But do LLMs have memory? Based on current practice, LLMs do appear to exhibit memory. So, what is the underlying mechanism of this memory? Previous research lacked a deep exploration of LLMs' memory capabilities and the underlying theory. In this paper, we use the Universal Approximation Theorem (UAT) to explain the memory mechanism in LLMs. We also conduct experiments to verify the memory capabilities of various LLMs, proposing a new method to assess their abilities based on the memory ability. We argue that LLM memory operates like Schrödinger's memory, meaning that it only becomes observable when a specific memory is queried. We can only determine if the model retains a memory based on its output in response to the query; otherwise, it remains indeterminate. Finally, we expand on this concept by comparing the memory capabilities of the human brain and LLMs, highlighting the similarities and differences in their operational mechanisms.
BibTeX
@misc{wang2024schrodinger,
  author = {Wei Wang and Qing Li},
  title = {{Schrodinger's Memory: Large Language Models}},
  year = {2024},
  eprint = {2409.10482},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10482},
}
Figure