Dynamic Weight Materialization to Bypass the Memory Wall

33 concepts23 papers1 proposal

In modern deep learning, the primary bottleneck for inference and training is often not the speed of the processor, but the cost of moving data. The "memory wall"—the growing disparity between high-speed computation and relatively slow, energy-intensive memory access—limits the deployment of large models on edge devices and high-performance accelerators alike.

A growing body of research proposes a fundamental shift in how we handle model parameters: instead of storing and retrieving full weight matrices, systems should store a compact representation and materialize weights on-the-fly using local computation. By trading "cheap" idle compute cycles for "expensive" memory bandwidth, these methods enable the execution of models that are orders of magnitude larger than the available physical memory.

Synthesis from Deterministic Seeds

The most extreme form of weight materialization involves replacing stored parameters with deterministic generation functions. If a layer's weights do not need to be learned—or can be represented as a transformation of a fixed distribution—they can be reconstructed using a pseudo-random number generator (PRNG) and a tiny fixed seed.

Hersche et al. demonstrate this for brain-computer interfaces, where large projection matrices are generated on-the-fly from seeds to perform high-dimensional feature expansion on microcontrollers. Similarly, Mordido et al. show that 1x1 convolutions in speech models can be replaced by fixed, sparse ternary matrices reconstructed via hash functions, eliminating the need to store millions of parameters.

For learned models, Shafipour et al. introduce SeedLM, which compresses Large Language Model (LLM) weights into seeds for hardware-efficient Linear Feedback Shift Registers. During inference, these seeds are expanded into weight blocks in real-time, effectively substituting DRAM traffic with on-chip logic. This concept extends to training as well; Malladi et al. utilize random seeds to generate weight perturbations for zeroth-order optimization, allowing massive models to be fine-tuned with the same memory footprint as inference.

Structural Reconstruction from Compressed Bases

Rather than generating weights from scratch, another approach decomposes weight matrices into a small set of basis vectors and a larger set of low-precision coefficients. The full weights are then reconstructed in situ using simple arithmetic.

  • Shift-and-Add Logic: Zhao et al. propose SmartExchange, which decomposes weights into a basis matrix and a sparse power-of-2 coefficient matrix. A custom accelerator reconstructs the weights using energy-efficient shift-and-add operations, bypassing the need for high-precision memory transfers.
  • Orthogonal Basis Vectors: Venieris et al. and Venieris et al. utilize Orthogonal Variable Spreading Factor (OVSF) codes to synthesize weights on-chip. By storing only the coefficients and generating the full tensors at runtime, their unzipFPGA framework significantly reduces off-chip memory traffic.
  • Low-Rank Components: Zhu et al. apply this to photonic accelerators, where computation is so fast that memory access becomes the absolute bottleneck. They reconstruct weights from low-rank bases to ensure the hardware is never starved for data.

Transient Materialization and Layer-wise Decompression

For models that are too large to fit in RAM even in a compressed state, researchers have developed "just-in-time" memory management. In this paradigm, weights are treated as transient objects that exist only for the duration of a specific computation.

Hao et al. and Yang et al. exploit the fact that floating-point exponents in neural networks are highly concentrated. Their frameworks maintain only compressed exponents in permanent memory, decompressing individual layers into a shared temporary buffer only when needed. Han et al. formalize this with PIPELOAD, a pipeline mechanism that evicts weights from RAM immediately after their layer's forward pass is complete, enabling Transformer models to run on devices with 90% less memory than typically required.

Similarly, Putzky et al. integrate parallelized entropy decoders directly into the GPU inference pipeline. By decompressing weights block-by-block during the kernel execution, they trade idle GPU compute cycles for a reduction in the total memory footprint.

Generative Weight Synthesis

A more complex strategy involves using a secondary "hypernetwork" to generate the weights of a primary model. This allows for "soft weight-sharing," where a small parameter set can represent a much larger, more expressive architecture.

Shih et al. introduce HyperSPNs, where a small neural network generates the weights for a large Probabilistic Circuit. By materializing these weights dynamically during the circuit traversal, the authors achieve a memory footprint that scales logarithmically with the input size. Abbasi et al. extend this to deep networks with PRANC, which reparameterizes a model as a linear combination of frozen, randomly initialized basis networks. Only the mixture coefficients are learned, and the basis components are generated on-the-fly, reducing peak memory usage during both training and inference.

Why It Works: The Efficiency Trade-off

The success of on-the-fly reconstruction rests on a simple economic reality of hardware: logic is cheaper than wires. As Venieris et al. argue, utilizing abundant on-chip logic to synthesize weights allows accelerators to prioritize limited memory bus capacity for non-compressible data like activations. Whether through PRNG seeds, basis decomposition, or entropy decoding, these methods all share a common goal: transforming the problem of storage into a problem of calculation, effectively bypassing the memory wall.

Key Foundational Work

ADIC: Anomaly Detection Integrated Circuit in 65-nm CMOS Utilizing Approximate Computing (2020) — contributed 2 concept elements to this direction.
Binarization Methods for Motor-Imagery Brain-Computer Interface Classification (2020) — contributed 2 concept elements to this direction.
SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_on_the_fly_weight_reconstruction_cd90bc34,
  title = {on-the-fly weight reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/on-the-fly-weight-reconstruction/rdr_cd90bc34e15e464fa8922c05ade7cc01},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_on_the_fly_weight_reconstruction_cd90bc34_v1,
  title = {on-the-fly weight reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/on-the-fly-weight-reconstruction/rdr_cd90bc34e15e464fa8922c05ade7cc01/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake