A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference

Michael V. DeBole, Rathinakumar Appuswamy, Neil McGlohon, Brian Taba, Steven K. Esser, Filipp Akopyan, John V. Arthur, Arnon Amir, Alexander Andreopoulos, Peter J. Carlson, Andrew S. Cassidy, Pallab Datta, Myron D. Flickner, Rajamohan Gandhasri, Guillaume J. Garreau, Megumi Ito, Jennifer L. Klamo, Jeffrey A. Kusnitz, Nathaniel J. McClatchey, Jeffrey L. McKinstry, Tapan K. Nayak, Carlos Ortega Otero, Hartmut Penner, William P. Risk, Jun Sawada, Jay Sivagnaname, Daniel F. Smith, Rafael Sousa, Ignacio Terrizzano, Takanori Ueda, Trent Gray-Donald, David D. Cox, Dharmendra S. ModhaarXiv 2025 · DOI · arXiv

Introduction

As the demand for Large Language Models (LLMs) grows, the industry faces a critical challenge: the "memory wall." Traditional GPU-based architectures rely on frequent, energy-intensive data transfers between processors and off-chip DRAM. This bottleneck not only limits inference speed but also contributes to a massive energy footprint. IBM Research addresses this by scaling the NorthPole neural inference accelerator from a single-chip prototype into a fully integrated, rack-level system designed specifically for low-latency, energy-efficient LLM inference.

System Architecture and Vertical Integration

The NorthPole system achieves its performance through end-to-end vertical integration across hardware, model mapping, quantization, and software. At the core is the NorthPole chip, a 12nm processor featuring 22 billion transistors and 224 MB of on-chip SRAM. Crucially, the architecture eliminates off-chip DRAM during inference, keeping model weights and Key-Value (KV) caches entirely on-chip.

Physical hierarchy of the NorthPole system, scaling from a single inference chip to a 288-card rack integrated with IBM watsonx.

To scale this architecture, the researchers developed a hierarchical system:

  1. Hardware Scaling: 16 NorthPole cards are integrated into a 2U server node, with 18 nodes forming a complete 42U rack. This results in a 288-card system capable of 115 peta-ops of 4-bit performance.
  2. Model Mapping: To accommodate models exceeding a single chip's memory, the system employs pipeline parallelism (distributing transformer layers across chips) and tensor parallelism (splitting individual layers).
  3. Software Stack: The system utilizes a containerized pipeline with an OpenAI-compatible API, facilitating seamless integration into cloud environments like IBM watsonx.

Quantization Strategy: SiLQ

To fit high-parameter models into the limited on-chip SRAM, IBM introduced SiLQ (Simple Large Language Model Quantization-aware training). This algorithm compresses models from bfloat16 to 4-bit weights and 8-bit activations/caches. Unlike post-training quantization methods that often degrade model "intelligence," SiLQ ensures that the compressed models maintain high accuracy across diverse reasoning and linguistic tasks.

Performance Evaluation

The system was primarily evaluated using the IBM Granite-3.3-8b-instruct model. The results demonstrate that the NorthPole rack delivers deterministic, ultra-low latency performance. For the Granite-8B model supporting 28 simultaneous users, the system achieved an inter-token latency of just 2.8 ms. Remarkably, this performance is maintained regardless of sequence length due to the lack of complex memory hierarchies.

Accuracy comparison between 4-bit NorthPole (SiLQ) and bfloat16 across 19 LLM benchmarks, demonstrating negligible performance degradation.

Benchmark testing across 19 datasets from the Open LLM Leaderboard (including ARC, HellaSwag, and MMLU) confirmed that the 4-bit NorthPole model matched the accuracy of the original bfloat16 model, averaging approximately 56.6% across all tests. This proves that the massive efficiency gains do not come at the cost of model quality.

Conclusion and Future Outlook

The NorthPole rack system represents a shift toward sustainable AI. By delivering 115 peta-ops and 3.7 PB/s of aggregate memory bandwidth within a 30 kW air-cooled rack, it provides a viable alternative to power-hungry GPU clusters for enterprise "Small Language Models" (<10B parameters).

While the "all-on-chip" approach necessitates a trade-off between context length and user concurrency, it excels in high-speed, deterministic agentic workflows. Future work will focus on integrating the Granite-4.0 model family, potentially further optimizing the balance between model size, context window, and energy efficiency.

Summary updated

BibTeX
@misc{debole2025scalable,
  author = {Michael V. DeBole and Rathinakumar Appuswamy and Neil McGlohon and Brian Taba and Steven K. Esser and Filipp Akopyan and John V. Arthur and Arnon Amir and Alexander Andreopoulos and Peter J. Carlson and Andrew S. Cassidy and Pallab Datta and Myron D. Flickner and Rajamohan Gandhasri and Guillaume J. Garreau and Megumi Ito and Jennifer L. Klamo and Jeffrey A. Kusnitz and Nathaniel J. McClatchey and Jeffrey L. McKinstry and Tapan K. Nayak and Carlos Ortega Otero and Hartmut Penner and William P. Risk and Jun Sawada and Jay Sivagnaname and Daniel F. Smith and Rafael Sousa and Ignacio Terrizzano and Takanori Ueda and Trent Gray-Donald and David D. Cox and Dharmendra S. Modha},
  title = {{A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference}},
  year = {2025},
  eprint = {2511.15950},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.15950},
}
Figure