31.1 A 14.08-to-135.69Token/s ReRAM-on-Logic Stacked Outlier-Free Large-Language-Model Accelerator with Block-Clustered Weight-Compression and Adaptive Parallel-Speculative-Decoding

Pingcheng Dong, Yonghao Tan, Xuejiao Liu, Peng Luo, Yu Liu, Di Pang, Songchen Ma, Xijie Huang, Shih-Yang Liu, D. Zhang, Zhichao Lu, Luhong Liang, Chi-Ying Tsui, Fengbin Tu, Liang Zhao, Kwang-Ting Cheng

TL;DR

This paper introduces a novel LLM accelerator that significantly boosts inference speed and energy efficiency by combining a ReRAM-on-logic stacking architecture with advanced speculative decoding techniques. The design tackles key bottlenecks in LLM processing, particularly weight external memory access and inefficient token prediction, achieving impressive performance gains.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but suffer from slow inference speeds, primarily due to the token-by-token autoregressive decoding process. This process involves extensive weight external memory access (EMA), which creates a significant latency bottleneck, especially for resource-constrained edge devices. Speculative decoding (SD) offers a solution by using a smaller "draft" LLM (DLM) to predict multiple tokens ahead, which are then verified by a larger "target" LLM (TLM). However, SD itself introduces new challenges: 1) Low-bit quantization for the TLM, while reducing EMA, can degrade accuracy due to activation outliers. 2) Limited on-chip memory in edge devices forces frequent EMA for the DLM weights. 3) At longer draft lengths, a high rejection rate of predicted tokens by the TLM negates the benefits of SD. This paper aims to overcome these limitations to enable faster and more efficient LLM inference on edge hardware.

Methods

The authors propose a three-pronged approach:

  1. Local Rotation Unit (LRU) for Outlier-Free Quantization: To address accuracy degradation from low-bit quantization in the TLM, they introduce an LRU. This unit approximates global rotation using decomposed, low-depth Fast Walsh-Hadamard Transforms (FWHT). By performing rotation in two stages, it effectively removes activation outliers while minimizing area overhead compared to traditional deep FWHT arrays.
  2. ReRAM-Stacked Processing-Near-Memory (RS-PNM) with Blockwise Vector Quantization (BVQ): To mitigate DLM weight EMA and limited on-chip memory, they employ a ReRAM-on-logic stacking technology. This RS-PNM architecture integrates high-density ReRAM with logic. Coupled with a Blockwise Vector Quantization (BVQ) algorithm, DLM weights are clustered into codebooks stored in ReRAM. This allows for efficient retrieval and reconstruction of weights, reducing EMA.
  3. Adaptive Parallel Speculative Decoding (APSD) with Workload-Decoupled Out-of-Order Scheduler (WDOS): To tackle the issue of high token rejection rates at longer draft lengths, they propose APSD. This scheme dynamically switches between short-draft non-parallel drafting and long-draft parallel draft-and-verify based on TLM verification feedback. The WDOS further enhances this by decoupling workloads into parallel instruction queues, enabling efficient resource and bandwidth utilization during parallel draft-and-verify.

Datasets

The paper evaluates its LLM accelerator using the MT-Bench dataset for performance characterization. Specific LLM models mentioned include LLaMA-160M (as DLM) and LLaMA2-7B (as TLM).

Key Insights and Ideas

  • ReRAM-on-Logic Stacking is Crucial: The paper strongly demonstrates how integrating ReRAM directly onto the logic die is essential for overcoming the memory bandwidth and capacity limitations that plague LLM inference on edge devices.
  • Decomposing FWHT for Efficiency: The LRU's approach of breaking down complex FWHT operations into smaller, cascaded stages is a clever way to achieve outlier removal without the massive area penalty of traditional methods.
  • Adaptive Speculative Decoding Pays Off: The dynamic switching strategy in APSD is key to balancing the benefits of longer draft lengths (more accepted tokens) with the risks of high rejection rates, leading to more robust and efficient speculative decoding.
  • Out-of-Order Scheduling for Parallelism: Decoupling workloads and using out-of-order scheduling is vital for maximizing the effectiveness of parallel processing in SD, preventing resource contention and improving throughput.

Key Results

The proposed LLM accelerator, fabricated in 55nm, achieves:

  • Throughput: 14.08-to-135.69 tokens/s.
  • Speedup: 4.46-to-7.17x speedup over vanilla speculative decoding.
  • Energy Efficiency: 3.74-to-4.85x energy savings compared to a BF16 SD baseline.
  • Quantization Accuracy: Achieves perplexity comparable to SOTA W8A16 LLM accelerators with W4A8 quantization.
  • Memory Integration: Integrates 3.43MB SRAM and 8MB stacked ReRAM with 25.6GB/s bandwidth. A 4-chip system scales ReRAM to 32MB with 102.4GB/s bandwidth.
  • APSD Performance: Achieves 1.1-to-1.29x speedup and 10-to-14% reduction in rejected token ratio compared to previous parallel SD methods.

Limitations and Future Work

  • ReRAM Variability: While not explicitly detailed as a limitation, ReRAM technology inherently has variability, which is often managed through calibration and error correction. The paper implies their techniques handle this.
  • Model Specificity: The results are demonstrated on specific LLaMA models. Future work could explore the generalizability of these techniques to a wider range of LLM architectures and sizes.
  • Scalability: While a 4-chip system is presented, further exploration of scaling to larger multi-chip configurations for even larger models would be a natural next step.
  • Dynamic Adaptation Granularity: The adaptive strategy in APSD could potentially be refined further for even more granular control over drafting strategies.

Summary updated

Abstract

This work presents a 55nm speculative decoding-based LLM accelerator with bumpingbased face-to-face ReRAM-on-logic stacking technology. It features a local rotation unit for outlier-free low-bit quantization, a stacking-aware PNM architecture co-designed with blockwise vector quantization to reduce weight EMA overheads, and an adaptive parallel speculative decoding scheme with out-of-order scheduler for high resource and bandwidth utilization. Our chip achieves 14.08-to-135.69token/s and 4.46-to-7.17× speedup over vanilla speculative decoding.
BibTeX
@inproceedings{dong2026token,
  author = {Pingcheng Dong and Yonghao Tan and Xuejiao Liu and Peng Luo and Yu Liu and Di Pang and Songchen Ma and Xijie Huang and Shih-Yang Liu and D. Zhang and Zhichao Lu and Luhong Liang and Chi-Ying Tsui and Fengbin Tu and Liang Zhao and Kwang-Ting Cheng},
  title = {{31.1 A 14.08-to-135.69Token/s ReRAM-on-Logic Stacked Outlier-Free Large-Language-Model Accelerator with Block-Clustered Weight-Compression and Adaptive Parallel-Speculative-Decoding}},
  booktitle = {ISSCC},
  year = {2026},
  doi = {10.1109/isscc49663.2026.11409211},
}
Figure