Exstill: The Most Efficient Single-Chip Processor for the Next Token

Valavan Manohararajah, Vithursan ThangarasaarXiv 2026 · arXiv

TL;DR

Exstill proposes a paradigm shift in AI hardware by co-designing silicon and model architectures within a gate-level optimization loop. By fixing the model architecture in silicon and keeping all weights and KV caches on-die, their pre-silicon 5nm design achieves 110x faster decoding and 690x lower energy consumption than an NVIDIA B300 for latency-critical small-model workloads.

What Problem Does This Paper Address?

Modern AI is increasingly reliant on "crews" of small models (1B–4B parameters) that surround a massive frontier model. These small models handle safety screening (guardrails), speculative drafting, tool-calling (agents), and domain-specific reasoning.

The problem is the "Generality Tax." Current GPUs are designed to run any model, which necessitates layers of software abstraction, kernel launches, and instruction decoding. More critically, they must constantly fetch weights and Key-Value (KV) caches from external High-Bandwidth Memory (HBM). For a small model like Qwen3-1.7B, the actual arithmetic (the "floor") costs only 0.89 mJ per token, yet a state-of-the-art NVIDIA B300 spends up to 1,120x that amount. As logic improves faster than memory bandwidth and wiring, this efficiency gap is only widening.

Methods

The authors move away from general-purpose hardware toward a Model-Silicon Co-Design approach:

  1. Removing the Layers: They fix a specific model architecture into the silicon's datapath. This eliminates the need for software kernels or instruction scheduling—tokens go in, and tokens come out.
  2. Removing the Movement: The design keeps the model weights and the KV caches for every active conversation (up to 12 users) entirely on-die. Nothing crosses an external memory interface during inference.
  3. Gate-Level Search Loop: Instead of using high-level spreadsheets, the authors costed over 10 million design candidates using a gate-level flow (placed-and-routed circuits at 5nm). This allowed them to treat the model architecture itself as a design variable, optimizing it alongside the silicon to lower the theoretical arithmetic floor.
  4. Polyglot-DC: They introduce a family of models (1.7B and 4B) that use this fixed architecture. While the architecture is frozen in silicon, the weights remain loadable, allowing the same chip to be specialized for different roles (drafting, guards, agents) via distillation.

Datasets

The paper evaluates the Polyglot-DC family across a massive battery of benchmarks:

  • Speculative Decoding: SPEED-Bench.
  • General Reasoning & Coding: GSM8K, MATH-500, AIME 2025, HumanEval, MBPP, LiveCodeBench.
  • Chat & Instruction Following: MT-Bench, AlpacaEval, Arena-Hard v2.
  • Safety/Guardrails: ToxicChat, OpenAI Moderation, Aegis 1.0/2.0, HarmBench, WildGuardTest, and the newly introduced PHTest and OR-Bench for measuring false alarms.
  • Agentic Workflows: Berkeley Function Calling Leaderboard (BFCL) v3/v4, $\tau^2$-bench, and $\tau^3$-bench.
  • Financial Reasoning: A suite of 10 benchmarks including FinQA, TAT-QA, and FinanceBench.

Key Insights and Ideas

  • The Arithmetic Floor is the Only Real Limit: Every model has an inherent energy cost defined by its MAC (multiply-accumulate) operations. Efficiency is the ratio of what the hardware spends to this floor.
  • Batching is a Dial, Not a Remedy: Batching on GPUs improves throughput and energy-per-token, but it does so by making users wait longer. The "energy-delay product" remains nearly constant, meaning batching doesn't actually solve the underlying inefficiency.
  • Generality is the Primary Waste: Over 99.7% of the energy spent by a GPU on a small model is overhead from software layers and data movement, not math.
  • Co-Design Lowers the Floor: By optimizing the model architecture specifically for the silicon it runs on, Exstill lowered the arithmetic floor of a 1.7B model from 0.89 mJ to 0.29 mJ without sacrificing quality.

Key Results

  • Raw Performance: Compared to a B300 serving 12 users, the Exstill DC1 design decodes tokens 110x faster (50 $\mu$s vs 5.35 ms) at 690x lower energy.
  • Speculative Decoding: As a standalone drafter, Polyglot-DC1-Draft speeds up the decoding of large frontier models (up to 2.8T parameters) by up to 6.6x.
  • Guardrails: The 4B "Guard" model achieves safety F1 scores comparable to 12B and 20B models but provides a verdict in under 1 millisecond, roughly 39x faster than a B300.
  • Reasoning Latency: In financial and agentic tasks, reasoning turns that take seconds on a GPU fall to tens of milliseconds, making "chain-of-thought" reasoning feel instantaneous to the user.

Limitations and Future Work

  • Pre-Silicon Estimates: The results are based on high-fidelity gate-level simulations and simulations of the B300, as the Exstill chip has not yet been fabricated.
  • Fixed Architecture: While weights are loadable, the underlying transformer structure (e.g., number of heads, dimensions) is fixed in silicon. Significant shifts in model architecture would require a new chip.
  • Scope: This report focuses on data-center co-processors. The authors note that applying this approach to edge devices, robotics, and mobile remains future work.
  • Software Comparison: The GPU baselines use vLLM; the authors estimate that a perfectly hand-tuned GPU serving stack might narrow the performance gap by about half, though it wouldn't solve the HBM traffic bottleneck.

Summary updated

Abstract

Generating a token with a language model requires a fixed amount of arithmetic, whose energy sets a floor beneath any hardware. For Qwen3-1.7B at a 16K-token context, gate-level measurements at 5 nm place this floor at 0.89 mJ per token, yet an NVIDIA B300 spends 391-1,120x that. The excess is the price of generality, software layers, programmable hardware and moving weights and key-value (KV) caches from external memory, and it widens with each process generation. We present a co-design approach that brings token generation close to this floor and then lowers the floor itself, optimizing silicon and model together in a gate-level search loop. Fixing one model architecture in the datapath, with weights still loadable, and holding weights and every conversation's KV cache on-die, a pre-silicon 5 nm design brings an unchanged Qwen3-1.7B token to 3.4x its floor. Co-designing the model with the circuits at fixed quality lowers the floor to 0.29 mJ and the token to 0.52 mJ, 1.8x the new floor and less than the baseline model's arithmetic alone, while on-die capacity grows from one user to twelve. Against a measured B300 serving the same twelve users, the design decodes each user's tokens 110x faster at 690x lower energy. To show that one fixed architecture can serve, as a co-processor, the latency-critical workloads around a frontier model, we introduce Polyglot-DC (1.7B and 4B, each with its own design), specialized for each job by distillation and task-specific training into its own weights. As a drafter it speeds single-user speculative decoding of large open models by up to 6.6x at the maximum reasoning effort. Its guard returns multilingual verdicts in a millisecond or less, within a point of the strongest open guard. Its agent and financial-reasoning models lead comparable open models on established benchmarks, with a reasoning turn falling from seconds to tens of milliseconds.
BibTeX
@misc{manohararajah2026exstill,
  author = {Valavan Manohararajah and Vithursan Thangarasa},
  title = {{Exstill: The Most Efficient Single-Chip Processor for the Next Token}},
  year = {2026},
  eprint = {2611.18859},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18859},
}
Figure