Resistive Memory-based Neural Differential Equation Solver for Score-based Diffusion Model

Jichang Yang, Hegan Chen, Jia Chen, Songqi Wang, Shaocong Wang, Yifei Yu, Xi Chen, Bo Wang, Xinyuan Zhang, Binbin Cui, Yi Li, Ning Lin, Meng Xu, Xiaoxin Xu, Xiaojuan Qi, Zhongrui Wang, Xumeng Zhang, Dashan Shang, Han Wang, Qi Liu, Kwang-Ting Cheng, Ming LiuarXiv 2024 · arXiv · OpenReview

Abstract

Human brains image complicated scenes when reading a novel. Replicating this imagination is one of the ultimate goals of AI-Generated Content (AIGC). However, current AIGC methods, such as score-based diffusion, are still deficient in terms of rapidity and efficiency. This deficiency is rooted in the difference between the brain and digital computers. Digital computers have physically separated storage and processing units, resulting in frequent data transfers during iterative calculations, incurring large time and energy overheads. This issue is further intensified by the conversion of inherently continuous and analog generation dynamics, which can be formulated by neural differential equations, into discrete and digital operations. Inspired by the brain, we propose a time-continuous and analog in-memory neural differential equation solver for score-based diffusion, employing emerging resistive memory. The integration of storage and computation within resistive memory synapses surmount the von Neumann bottleneck, benefiting the generative speed and energy efficiency. The closed-loop feedback integrator is time-continuous, analog, and compact, physically implementing an infinite-depth neural network. Moreover, the software-hardware co-design is intrinsically robust to analog noise. We experimentally validate our solution with 180 nm resistive memory in-memory computing macros. Demonstrating equivalent generative quality to the software baseline, our system achieved remarkable enhancements in generative speed for both unconditional and conditional generation tasks, by factors of 64.8 and 156.5, respectively. Moreover, it accomplished reductions in energy consumption by factors of 5.2 and 4.1. Our approach heralds a new horizon for hardware solutions in edge computing for generative AI applications.

Introduction: The Challenge of Generative AI

The rapid advancement of AI-Generated Content (AIGC) has revolutionized how we create digital media. However, state-of-the-art models like score-based diffusion are computationally intensive. This intensity stems from the fundamental architecture of digital computers, where the physical separation of memory and processing (the von Neumann bottleneck) leads to significant energy and time overheads during the iterative denoising process. Furthermore, while the underlying mathematics of diffusion models are continuous neural differential equations (NDEs), digital systems must discretize these into many steps, further compounding inefficiencies.

Brain-Inspired Co-Design

To address these limitations, researchers have developed a hardware-software co-design inspired by the human brain's efficient "imagination" capabilities. Unlike digital computers that solve equations discretely, the proposed system uses an analog, time-continuous approach. By employing emerging resistive memory (RRAM) for in-memory computing (IMC), the system performs matrix multiplications directly within the memory array, effectively eliminating the data transfer bottleneck.

Conceptual comparison between human brain imagination, digital computer discrete solvers, and the proposed continuous analog co-design.

Methods: In-Memory Computing and Feedback Integration

The core of the solver is a closed-loop feedback integrator that physically implements an "infinite-depth" neural network. In this setup, weights are stored as conductance values in RRAM cells. The signal evolves continuously through the circuit, solving the differential equations in real-time without the need for a global clock or discrete steps. Interestingly, the inherent noise of analog hardware—typically seen as a drawback—is leveraged here to provide the stochasticity required for diverse image generation in score-based models.

Experimental Results and Performance

The system was tested on both unconditional generation and conditional latent diffusion using the EMNIST dataset. In the conditional task, a Variational Autoencoder (VAE) compressed images into a 2D latent space, which the RRAM solver then navigated to generate specific characters (H, K, U).

Experimental results for conditional latent diffusion on EMNIST, demonstrating significant speed and energy improvements over digital baselines.

The experimental results, obtained using a 180 nm RRAM process, are compelling. The analog solver achieved a 64.8x speedup for unconditional tasks and a 156.5x speedup for conditional tasks compared to traditional digital software baselines. Energy consumption was simultaneously reduced by factors of 5.2 and 4.1, respectively, while maintaining generative quality (measured by KL Divergence) comparable to high-precision digital methods.

Conclusion

This research demonstrates that analog in-memory computing can overcome the efficiency barriers of current generative AI. By aligning hardware architecture with the continuous nature of neural differential equations, this approach opens new possibilities for deploying powerful generative models on low-power edge devices, from wearables to autonomous sensors.

Summary updated

Abstract

Human brains image complicated scenes when reading a novel. Replicating this imagination is one of the ultimate goals of AI-Generated Content (AIGC). However, current AIGC methods, such as score-based diffusion, are still deficient in terms of rapidity and efficiency. This deficiency is rooted in the difference between the brain and digital computers. Digital computers have physically separated storage and processing units, resulting in frequent data transfers during iterative calculations, incurring large time and energy overheads. This issue is further intensified by the conversion of inherently continuous and analog generation dynamics, which can be formulated by neural differential equations, into discrete and digital operations. Inspired by the brain, we propose a time-continuous and analog in-memory neural differential equation solver for score-based diffusion, employing emerging resistive memory. The integration of storage and computation within resistive memory synapses surmount the von Neumann bottleneck, benefiting the generative speed and energy efficiency. The closed-loop feedback integrator is time-continuous, analog, and compact, physically implementing an infinite-depth neural network. Moreover, the software-hardware co-design is intrinsically robust to analog noise. We experimentally validate our solution with 180 nm resistive memory in-memory computing macros. Demonstrating equivalent generative quality to the software baseline, our system achieved remarkable enhancements in generative speed for both unconditional and conditional generation tasks, by factors of 64.8 and 156.5, respectively. Moreover, it accomplished reductions in energy consumption by factors of 5.2 and 4.1. Our approach heralds a new horizon for hardware solutions in edge computing for generative AI applications.
BibTeX
@misc{yang2024resistive,
  author = {Jichang Yang and Hegan Chen and Jia Chen and Songqi Wang and Shaocong Wang and Yifei Yu and Xi Chen and Bo Wang and Xinyuan Zhang and Binbin Cui and Yi Li and Ning Lin and Meng Xu and Yi Li and Xiaoxin Xu and Xiaojuan Qi and Zhongrui Wang and Xumeng Zhang and Dashan Shang and Han Wang and Qi Liu and Kwang-Ting Cheng and Ming Liu},
  title = {{Resistive Memory-based Neural Differential Equation Solver for Score-based Diffusion Model}},
  year = {2024},
  eprint = {2404.05648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05648},
}
Figure