Abstract
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit the challenges associated with interpretability of feed-forward modules (FFNs) and propose MemoryLLM, which aims to decouple FFNs from self-attention and enables us to study the decoupled FFNs as context-free token-wise neural retrieval memory. In detail, we investigate how input tokens access memory locations within FFN parameters and the importance of FFN memory across different downstream tasks. MemoryLLM achieves context-free FFNs by training them in isolation from self-attention directly using the token embeddings. This approach allows FFNs to be pre-computed as token-wise lookups (ToLs), enabling on-demand transfer between VRAM and storage, additionally enhancing inference efficiency. We also introduce Flex-MemoryLLM, positioning it between a conventional transformer design and MemoryLLM. This architecture bridges the performance gap caused by training FFNs with context-free token-wise embeddings.
Introduction
In standard Large Language Models (LLMs), self-attention and Feed-Forward Networks (FFNs) are tightly coupled within the residual stream. While self-attention mechanisms have been extensively analyzed, FFNs—which typically account for approximately two-thirds of a model's parameters—remain largely opaque.
The primary interpretability challenge stems from the fact that FFNs consume a dynamic, non-interpretable additive mixture of the residual stream and self-attention outputs. This context-dependent input makes it difficult to map FFN activations back to specific input tokens. Moreover, the sheer size of these parameters creates significant VRAM bottlenecks during inference. MemoryLLM addresses these issues by disentangling FFNs from self-attention to create a deterministic, interpretable memory mapped to a finite vocabulary.
Architectural Innovation: Decoupling FFNs
MemoryLLM simplifies the conventional Transformer block by decoupling the FFN from the contextualized residual stream. In this architecture, FFNs read only the static, context-free token embeddings ($x_0$) from the input layer, while self-attention continues to operate on the residual stream in parallel.
To bridge the potential performance gap of purely context-free FFNs, the authors also introduce Flex-MemoryLLM. This hybrid design splits the FFN parameter budget: a small dense module (FFN-C) processes the contextualized residual stream, while the larger portion (FFN-M) acts as the context-free, offloadable memory.
Pre-computed Token-wise Lookups (ToLs)
A key advantage of the MemoryLLM design is that FFN outputs depend solely on the input token ID. This allows for the pre-computation of FFN outputs for the entire vocabulary offline, resulting in Token-wise Lookups (ToLs).
During inference, these ToLs can be stored on disk and loaded into VRAM on-demand. This "Plug-n-Play" capability significantly reduces active memory requirements. By leveraging Zipf's Law—caching ToLs for the most frequent tokens in VRAM while keeping others on disk—the model can maintain high performance with a fraction of the active parameters.
Key Insights and Results
The authors trained models from scratch on the C4 dataset at scales up to 1B parameters and evaluated them across various downstream tasks.
Interpretability and Redundancy
- Semantic Mapping: Clustering memory activation coefficients revealed that semantically similar tokens (e.g., punctuation, names, locations) activate the same memory keys, confirming that FFNs act as structured knowledge repositories.
- Functional Specialization: FFNs were found to be dominant in recall and retrieval tasks (e.g., Wikitext-2, LAMBDA), whereas tasks requiring logical reasoning (e.g., HellaSwag) were less dependent on FFN contributions.
- Layer Redundancy: Analysis showed that FFN memory contribution drops significantly after the first few layers. Middle-layer ToLs are highly redundant, suggesting that dropping them has minimal impact on performance.
Performance and Efficiency
- VRAM Efficiency: MemoryLLM allows offloading approximately two-thirds of parameters. A 1B parameter model requires only ~402M active parameters in VRAM.
- Inference Speed: Despite the storage overhead, a 1B MemoryLLM achieved a decoding speed of 14.42 ms/token, outperforming the baseline dense model's 21.50 ms/token due to reduced active computation.
- Robustness: Flex-MemoryLLM (704M active parameters) matches the performance of a standard 1.2B parameter model and significantly outperforms models pruned to the same active parameter count using methods like SparseGPT or Wanda.
Conclusion
MemoryLLM provides a novel framework for making FFNs interpretable and efficient. By transforming FFNs into pre-computable lookups, it enables significant VRAM savings and faster inference without sacrificing the model's ability to store factual knowledge. Future work will focus on compressing ToLs to reduce storage overhead and further exploring the dynamics of FFN over-parameterization.
Summary updated