In conventional computer architectures, processing units (like CPUs and GPUs) are physically separated from memory storage units (like SRAM and DRAM). To perform even a simple addition, data must travel from memory, across a communication bus, into a processor register, and then back to memory.
This constant shuttling of data creates a massive energy and latency penalty known as the von Neumann bottleneck. As Vivienne Sze et al. point out in Hardware for machine learning: Challenges and opportunities, data movement—rather than the actual computation itself—is the primary driver of energy consumption in modern machine learning workloads. For power-constrained edge devices or massive large language models, this "data tax" imposes a strict efficiency wall. Lukas Cavigelli et al. demonstrate in Hyperdrive: A Multi-Chip Systolically Scalable Binary-Weight CNN Inference Engine that the energy required to move data between external memory and the processing core significantly exceeds the energy used for the mathematical operations themselves.
To break through this wall, researchers are developing In-Memory Computing (IMC) and Processing-in-Memory (PIM) architectures. Instead of moving data to the processor, these systems perform computations directly inside the memory array itself, eliminating the communication bus overhead entirely.
The Intuition: Computing with Physical Laws
To understand why In-Memory Computing is so efficient, consider the core mathematical operation of modern artificial intelligence: the vector-matrix multiplication (VMM). In a standard digital processor, computing a VMM requires fetching weights one by one, multiplying them by inputs, and accumulating the results in a digital register.
IMC architectures replace this sequential digital process with parallel analog physics. By organizing memory cells into a crossbar array, the system can exploit Ohm’s Law and Kirchhoff’s Current Law to compute an entire VMM in a single step:
- Weights as Conductances: The neural network weights are programmed into the memory cells as physical electrical conductances ($G$).
- Inputs as Voltages: The inputs to the layer are applied simultaneously as voltages ($V$) across the rows of the array.
- Instant Multiplication: According to Ohm's Law ($I = V \times G$), the current flowing through each memory cell is naturally proportional to the product of the input voltage and the stored weight.
- Instant Accumulation: According to Kirchhoff's Current Law, the currents from all cells along a single column naturally sum together at the bottom of the wire.
As Fei Wang et al. describe in In-memory search with learning to hash based on resistive memory for recommendation acceleration, this physical integration allows massive parallel computations to occur exactly where the data resides, fundamentally bypassing the data tax.
Diverse Physical Substrates
This concept can be implemented across a variety of memory technologies, each offering unique trade-offs for different applications:
- SRAM: Standard silicon memory can be modified for IMC. For instance, Sumon Kumar Bose et al. developed a custom architecture in A 75kb SRAM in 65nm CMOS for In-Memory Computing Based Neuromorphic Image Denoising that performs localized image filtering directly inside the SRAM, achieving massive energy savings for neuromorphic vision processing.
- Phase-Change Memory (PCM): Non-volatile PCM devices alter their physical state between amorphous and crystalline forms to vary conductance. Geethan Karunaratne et al. utilized hundreds of thousands of PCM devices in In-memory hyperdimensional computing to execute highly parallel, energy-efficient classification tasks.
- Memristors and Resistive RAM (RRAM): These emerging devices change their resistance based on the history of applied voltage. Can Li et al. demonstrated in Efficient and self-adaptive in-situ learning in multilayer memristor neural networks that memristors can be monolithically integrated with standard CMOS arrays to enable "in-situ" learning, allowing neural networks to train directly on the hardware substrate.
- Processing-Near-Memory (PNM): For workloads too massive to fit entirely inside analog crossbars—such as long-context large language models—researchers move the processing elements immediately adjacent to the memory modules using high-speed interconnects. Dowon Kim et al. applied this in Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits to offload Key-Value cache management directly to memory modules, drastically reducing GPU data movement overhead.
Mitigating Analog Imperfections
If In-Memory Computing is so powerful, why hasn't it completely replaced digital processors? The primary challenge stems from the messy reality of analog physics. Physical memory devices suffer from non-idealities: device-to-device manufacturing variability, thermal noise, and conductance drift over time.
To achieve software-equivalent accuracy on analog hardware, researchers rely on two core strategies:
Mixed-Precision Architectures
Instead of forcing the entire system into the analog domain, mixed-precision systems combine the best of both worlds. They perform the heavy, repetitive matrix multiplications inside analog memory arrays while maintaining high-precision digital memory for tasks sensitive to noise, such as weight accumulation during training. As Manuel Le Gallo et al. demonstrate in Mixed-Precision Deep Learning Based on Computational Memory, this hybrid approach preserves the strict accuracy requirements of deep learning while retaining the massive energy efficiency gains of analog computing.
Hardware-Aware Training
Another approach is to make the neural network models resilient to hardware imperfections before they are ever deployed. By modeling device noise, non-linearities, and drift during the software training phase, the network learns to find parameter spaces that are inherently robust to analog faults.
To facilitate this, simulation frameworks like MemTorch, introduced by Corey Lammie et al. in MemTorch: An Open-source Simulation Framework for Memristive Deep Learning Systems, allow developers to inject real-world hardware non-idealities directly into the deep learning design pipeline. Using these hardware-aware methodologies, Katie Spoon et al. showed in Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices that complex models like Transformers can achieve near-lossless digital accuracy even when running on noisy, drifting analog memory devices.