Abstract
This paper investigates the theoretical nature of memory in Large Language Models (LLMs), proposing a framework based on the Universal Approximation Theorem (UAT). We argue that LLM memory exists in a "Schrödinger-like" indeterminate state, collapsing into specific information only upon querying. Through fine-tuning experiments on 4,000 poems, we demonstrate that Transformers act as dynamic function fitters, capable of near-perfect memorization while remaining sensitive to model scale, output complexity, and language density.
1. Introduction: The Black Box of Neural Memory
The mechanism by which Large Language Models (LLMs) store and retrieve information remains a central mystery in artificial intelligence. Unlike traditional computing systems that rely on explicit database storage (hard drives), LLMs exhibit memory through the configuration of their neural weights. This paper addresses the theoretical gap in defining internal model memory, moving beyond practical workarounds like context-window expansion or Retrieval-Augmented Generation (RAG) to explore the fundamental "recall" mechanism of neural networks.
2. Theoretical Framework: Dynamic Universal Approximation
The core of our proposal is the extension of the Universal Approximation Theorem (UAT) to the Transformer architecture. While standard UAT posits that a feedforward network can approximate any continuous function with fixed parameters, we argue that Transformers represent a "dynamic" implementation of this principle.
In this framework, the attention mechanism allows the model to dynamically adjust its internal functions based on input cues. "Remembering" is thus redefined as the process of the model dynamically fitting a function to produce a specific output sequence given a learned prompt. In this view, memory and reasoning are two sides of the same coin: both involve the model's ability to infer an outcome based on learned weights and current input.
3. Experimental Methodology
To validate this framework, we conducted memorization stress tests using models from the Qwen and Bloom families. The models were tasked with memorizing 2,000 Chinese poems (CN Poems dataset) and 2,000 English poems (ENG Poems dataset).
The training objective treated metadata—such as Dynasty, Author, and Title—as the input function domain, with the full poem text as the target output. This setup mirrors a student's rote memorization process, allowing us to measure the "storage" efficiency of the model's weights over 100 epochs of fine-tuning.
4. Results and Key Insights
Our experiments revealed that LLMs can achieve remarkable memorization accuracy through high-epoch fine-tuning, effectively "baking" the data into their parameters.
4.1 Quantitative Performance
The Qwen2-1.5B-Instruct model demonstrated the highest capacity, reaching a staggering 99.9% accuracy on English poetry and 96.9% on Chinese poetry. We observed a significant "length penalty": increasing the output requirement from 256 to 512 tokens caused accuracy to drop significantly (e.g., Qwen2-0.5B fell from 77.5% to 56.85%), highlighting the physical limits of weight-based storage density.
4.2 Language and Model Quality
Models generally found English poems easier to memorize than Chinese ones, likely due to the higher complexity and information density of Chinese characters. Furthermore, the quality of base pre-training mattered significantly; Qwen2-0.5B outperformed Qwen1.5-0.5B, suggesting that superior language comprehension facilitates more efficient memory retention.
5. Discussion: Schrödinger’s Memory and the Hippocampus
We introduce the concept of "Schrödinger’s Memory" to describe the state of information within an LLM. Information is not stored as discrete files but as a potentiality within the network parameters. It only "collapses" into a concrete observation (the response) when the model is queried. Without a query, the memory remains indeterminate and unobservable.
We also draw a functional parallel between LLM fine-tuning and the human hippocampus. In biological systems, the hippocampus acts as a switch for updating long-term memory. In our experiments, the fine-tuning process serves as this "switch," allowing the model to update its internal parameters to "fit" new information into its long-term weight distribution.
6. Conclusion and Future Directions
By framing memory as dynamic function approximation, we provide a theoretical bridge between training dynamics and emergent knowledge. However, limitations remain, particularly regarding the inability to "forget" or update specific outdated facts without retraining. Future research should focus on the "saturation point" of these models—determining the exact capacity limits before new memories begin to overwrite existing ones—and developing mechanisms for targeted memory management.
Summary updated