TL;DR
Exstill proposes a paradigm shift in AI hardware by co-designing silicon and model architectures within a gate-level optimization loop. By fixing the model architecture in silicon and keeping all weights and KV caches on-die, their pre-silicon 5nm design achieves 110x faster decoding and 690x lower energy consumption than an NVIDIA B300 for latency-critical small-model workloads.
What Problem Does This Paper Address?
Modern AI is increasingly reliant on "crews" of small models (1B–4B parameters) that surround a massive frontier model. These small models handle safety screening (guardrails), speculative drafting, tool-calling (agents), and domain-specific reasoning.
The problem is the "Generality Tax." Current GPUs are designed to run any model, which necessitates layers of software abstraction, kernel launches, and instruction decoding. More critically, they must constantly fetch weights and Key-Value (KV) caches from external High-Bandwidth Memory (HBM). For a small model like Qwen3-1.7B, the actual arithmetic (the "floor") costs only 0.89 mJ per token, yet a state-of-the-art NVIDIA B300 spends up to 1,120x that amount. As logic improves faster than memory bandwidth and wiring, this efficiency gap is only widening.
Methods
The authors move away from general-purpose hardware toward a Model-Silicon Co-Design approach:
- Removing the Layers: They fix a specific model architecture into the silicon's datapath. This eliminates the need for software kernels or instruction scheduling—tokens go in, and tokens come out.
- Removing the Movement: The design keeps the model weights and the KV caches for every active conversation (up to 12 users) entirely on-die. Nothing crosses an external memory interface during inference.
- Gate-Level Search Loop: Instead of using high-level spreadsheets, the authors costed over 10 million design candidates using a gate-level flow (placed-and-routed circuits at 5nm). This allowed them to treat the model architecture itself as a design variable, optimizing it alongside the silicon to lower the theoretical arithmetic floor.
- Polyglot-DC: They introduce a family of models (1.7B and 4B) that use this fixed architecture. While the architecture is frozen in silicon, the weights remain loadable, allowing the same chip to be specialized for different roles (drafting, guards, agents) via distillation.
Datasets
The paper evaluates the Polyglot-DC family across a massive battery of benchmarks:
- Speculative Decoding: SPEED-Bench.
- General Reasoning & Coding: GSM8K, MATH-500, AIME 2025, HumanEval, MBPP, LiveCodeBench.
- Chat & Instruction Following: MT-Bench, AlpacaEval, Arena-Hard v2.
- Safety/Guardrails: ToxicChat, OpenAI Moderation, Aegis 1.0/2.0, HarmBench, WildGuardTest, and the newly introduced PHTest and OR-Bench for measuring false alarms.
- Agentic Workflows: Berkeley Function Calling Leaderboard (BFCL) v3/v4, $\tau^2$-bench, and $\tau^3$-bench.
- Financial Reasoning: A suite of 10 benchmarks including FinQA, TAT-QA, and FinanceBench.
Key Insights and Ideas
- The Arithmetic Floor is the Only Real Limit: Every model has an inherent energy cost defined by its MAC (multiply-accumulate) operations. Efficiency is the ratio of what the hardware spends to this floor.
- Batching is a Dial, Not a Remedy: Batching on GPUs improves throughput and energy-per-token, but it does so by making users wait longer. The "energy-delay product" remains nearly constant, meaning batching doesn't actually solve the underlying inefficiency.
- Generality is the Primary Waste: Over 99.7% of the energy spent by a GPU on a small model is overhead from software layers and data movement, not math.
- Co-Design Lowers the Floor: By optimizing the model architecture specifically for the silicon it runs on, Exstill lowered the arithmetic floor of a 1.7B model from 0.89 mJ to 0.29 mJ without sacrificing quality.
Key Results
- Raw Performance: Compared to a B300 serving 12 users, the Exstill DC1 design decodes tokens 110x faster (50 $\mu$s vs 5.35 ms) at 690x lower energy.
- Speculative Decoding: As a standalone drafter, Polyglot-DC1-Draft speeds up the decoding of large frontier models (up to 2.8T parameters) by up to 6.6x.
- Guardrails: The 4B "Guard" model achieves safety F1 scores comparable to 12B and 20B models but provides a verdict in under 1 millisecond, roughly 39x faster than a B300.
- Reasoning Latency: In financial and agentic tasks, reasoning turns that take seconds on a GPU fall to tens of milliseconds, making "chain-of-thought" reasoning feel instantaneous to the user.
Limitations and Future Work
- Pre-Silicon Estimates: The results are based on high-fidelity gate-level simulations and simulations of the B300, as the Exstill chip has not yet been fabricated.
- Fixed Architecture: While weights are loadable, the underlying transformer structure (e.g., number of heads, dimensions) is fixed in silicon. Significant shifts in model architecture would require a new chip.
- Scope: This report focuses on data-center co-processors. The authors note that applying this approach to edge devices, robotics, and mobile remains future work.
- Software Comparison: The GPU baselines use vLLM; the authors estimate that a perfectly hand-tuned GPU serving stack might narrow the performance gap by about half, though it wouldn't solve the HBM traffic bottleneck.
Summary updated