Bypassing the Von Neumann Bottleneck with In-Memory Computing

431 concepts370 papers10 proposals

In conventional computer architectures, processing units (like CPUs and GPUs) are physically separated from memory storage units (like SRAM and DRAM). To perform even a simple addition, data must travel from memory, across a communication bus, into a processor register, and then back to memory.

This constant shuttling of data creates a massive energy and latency penalty known as the von Neumann bottleneck. As Vivienne Sze et al. point out in Hardware for machine learning: Challenges and opportunities, data movement—rather than the actual computation itself—is the primary driver of energy consumption in modern machine learning workloads. For power-constrained edge devices or massive large language models, this "data tax" imposes a strict efficiency wall. Lukas Cavigelli et al. demonstrate in Hyperdrive: A Multi-Chip Systolically Scalable Binary-Weight CNN Inference Engine that the energy required to move data between external memory and the processing core significantly exceeds the energy used for the mathematical operations themselves.

To break through this wall, researchers are developing In-Memory Computing (IMC) and Processing-in-Memory (PIM) architectures. Instead of moving data to the processor, these systems perform computations directly inside the memory array itself, eliminating the communication bus overhead entirely.

The Intuition: Computing with Physical Laws

To understand why In-Memory Computing is so efficient, consider the core mathematical operation of modern artificial intelligence: the vector-matrix multiplication (VMM). In a standard digital processor, computing a VMM requires fetching weights one by one, multiplying them by inputs, and accumulating the results in a digital register.

IMC architectures replace this sequential digital process with parallel analog physics. By organizing memory cells into a crossbar array, the system can exploit Ohm’s Law and Kirchhoff’s Current Law to compute an entire VMM in a single step:

  1. Weights as Conductances: The neural network weights are programmed into the memory cells as physical electrical conductances ($G$).
  2. Inputs as Voltages: The inputs to the layer are applied simultaneously as voltages ($V$) across the rows of the array.
  3. Instant Multiplication: According to Ohm's Law ($I = V \times G$), the current flowing through each memory cell is naturally proportional to the product of the input voltage and the stored weight.
  4. Instant Accumulation: According to Kirchhoff's Current Law, the currents from all cells along a single column naturally sum together at the bottom of the wire.

As Fei Wang et al. describe in In-memory search with learning to hash based on resistive memory for recommendation acceleration, this physical integration allows massive parallel computations to occur exactly where the data resides, fundamentally bypassing the data tax.

Diverse Physical Substrates

This concept can be implemented across a variety of memory technologies, each offering unique trade-offs for different applications:

Mitigating Analog Imperfections

If In-Memory Computing is so powerful, why hasn't it completely replaced digital processors? The primary challenge stems from the messy reality of analog physics. Physical memory devices suffer from non-idealities: device-to-device manufacturing variability, thermal noise, and conductance drift over time.

To achieve software-equivalent accuracy on analog hardware, researchers rely on two core strategies:

Mixed-Precision Architectures

Instead of forcing the entire system into the analog domain, mixed-precision systems combine the best of both worlds. They perform the heavy, repetitive matrix multiplications inside analog memory arrays while maintaining high-precision digital memory for tasks sensitive to noise, such as weight accumulation during training. As Manuel Le Gallo et al. demonstrate in Mixed-Precision Deep Learning Based on Computational Memory, this hybrid approach preserves the strict accuracy requirements of deep learning while retaining the massive energy efficiency gains of analog computing.

Hardware-Aware Training

Another approach is to make the neural network models resilient to hardware imperfections before they are ever deployed. By modeling device noise, non-linearities, and drift during the software training phase, the network learns to find parameter spaces that are inherently robust to analog faults.

To facilitate this, simulation frameworks like MemTorch, introduced by Corey Lammie et al. in MemTorch: An Open-source Simulation Framework for Memristive Deep Learning Systems, allow developers to inject real-world hardware non-idealities directly into the deep learning design pipeline. Using these hardware-aware methodologies, Katie Spoon et al. showed in Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices that complex models like Transformers can achieve near-lossless digital accuracy even when running on noisy, drifting analog memory devices.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_von_neumann_bottleneck_data_movement_ene_8de701b9,
  title = {von Neumann bottleneck (data movement energy overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/von-neumann-bottleneck-data-movement-energy-overhead/rdr_8de701b9aa0144eea159cfdf45273c54},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_von_neumann_bottleneck_data_movement_ene_8de701b9_v1,
  title = {von Neumann bottleneck (data movement energy overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/von-neumann-bottleneck-data-movement-energy-overhead/rdr_8de701b9aa0144eea159cfdf45273c54/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake