A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights

Tanmay SaharXiv 2026 · arXiv

TL;DR

Researchers have developed a way to "graft" exact digital circuits—like multi-digit arithmetic and Boolean logic—directly into the weights of pretrained LLMs without any training or fine-tuning. By using analytical "zero-gradient" primitives, they achieve near-perfect mathematical accuracy while preserving the model's original language abilities, effectively embedding a deterministic CPU inside a probabilistic neural network.

What Problem Does This Paper Address?

Despite their impressive reasoning, Large Language Models (LLMs) are notoriously brittle when it comes to exact multi-digit arithmetic. They often hallucinate digits or fail on simple carries because they are probabilistic next-token predictors, not deterministic calculators.

Current solutions generally fall into two camps:

  1. Fine-tuning (e.g., LoRA): Training the model on math problems. This is data-intensive, can lead to "catastrophic forgetting" of language skills, and often fails to generalize to larger numbers.
  2. External Tools: Routing math queries to a Python interpreter. This requires pausing the model's generation, external infrastructure, and complex "tool-use" prompting.

The authors tackle the challenge of embedding exact, deterministic logic directly into the model's weights without needing a single training example or an external interpreter, all while ensuring the model doesn't lose its "natural language" touch.

Methods

The core innovation is a set of zero-gradient circuit-grafting primitives. Instead of training, the authors use mathematical "closed-form" solutions to modify specific weights.

  • Mellin-GLU Quadrature: Normalization layers like RMSNorm are usually a headache for exact computation because they scale activations by an unknown factor. The authors use a "log-spaced gate grid" to turn this unknown scaling into a simple translation along a grid, allowing for precise multiplication and addition.
  • Indefinite-RoPE & GatedDeltaNet Shifts: These are used for "routing." By carefully setting weights in the attention mechanism (specifically using signed Chebyshev weights), they can "park" inactive queries and precisely move data between tokens without interference.
  • Axis-Aligned Width Extension ($d \to d+K$): Rather than trying to squeeze new logic into the existing "brain" of the LLM, the authors add a small number of new "wires" (dimensions) to the model's internal representation. This "sidecar" approach prevents the new logic from corrupting the model's original knowledge.
  • Automated Compiler: They built a 5-stage compiler that takes a high-level description of a circuit (like an 8-digit multiplier) and automatically generates the weight patches for models like IBM Granite or Alibaba Qwen.
Geometric intuition for normalization cancellation and self-sink routing.

Datasets

The authors evaluated their "grafted" models on several benchmarks:

  • GSM8K: A standard benchmark for grade-school math word problems.
  • Tango: A newly introduced, pre-registered suite of 4,392 splits designed to test long-context reasoning and ensure no degradation in natural language performance.
  • NI-8: An 8-task natural language benchmark to verify "noninferiority" (making sure the model is still a good LLM).
  • OOD Triggers: Over 11,000 "out-of-domain" examples were used to ensure the math circuits only fire when they are supposed to, avoiding "false triggers" during normal conversation.

Key Insights and Ideas

  • Normalization as a Signal: Instead of fighting RMSNorm, the authors realized that its scaling factor can be treated as a coordinate on a log-grid. This allows them to perform "scale-invariant" math.
  • The "Sidecar" Efficiency: By implementing the circuits as a "factorized block-sparse sidecar," they drastically reduced the memory overhead. On an A100 GPU, this approach uses significantly less VRAM than a full dense implementation of the same logic.
  • Width vs. Projection: A major "aha" moment was proving that adding new dimensions (width extension) is far superior to trying to "carve out" space in existing dimensions using techniques like SVD. As shown in Figure 3, the "width extension" approach results in orders of magnitude less disruption to the model's original language performance (measured by KL divergence).
Figure 3. Subspace geometry and decomposed capability profile (BF16). (a) Zero-gradient SVD projection (P⊥) incurs 43×– 3,800× higher next-token KL than orthogonal width extension (K = 1,140), while uncompensated d + K (M3) inflates KL by 547×. (b) Canonical grafts reach 97.08%–99.50% leading-integer match, 100% blind ALU-prefix exactness, 98%–100% inline resumption, and paired Tango noninferiority on GSM8K-200 and NI-8 (N = 2,350).

Key Results

  • Near-Perfect Math: On 6-digit arithmetic, the grafted Alibaba Qwen 3.5 4B reached 100.00% accuracy, while IBM Granite 4.2 3B reached 99.17%. This significantly outperformed traditional LoRA fine-tuning, which struggled to reach even 62% accuracy on the same tasks.
  • Zero Language Degradation: The models passed all 14/14 pre-registered Tango splits, proving that the math "CPU" doesn't interfere with the model's ability to write prose or follow instructions.
  • Seamless Resumption: The models showed 98%–100% "mid-sentence resumption" accuracy, meaning they can switch from math back to natural language fluently.
  • Robustness: The system had zero false triggers across over 10,000 diverse non-math prompts.

Limitations and Future Work

  • BPE Alignment: The biggest remaining hurdle is the model's tokenizer. If a model's BPE tokenizer merges digits in a way the circuit doesn't expect (e.g., merging "12" into one token but "3" into another), it can cause "mis-reads." The authors suggest "Base-1000 canonical BPE emission" as a fix.
  • Complexity Limits: While they successfully implemented 8-digit multiplication and division, extremely complex algorithms might require more layers than are available in smaller models.
  • Security Auditing: The authors note a "dual-use" risk: these same primitives could be used to graft "backdoors" or hidden payloads into open-weight models that are nearly impossible to detect with standard behavioral benchmarks. Future work will need to focus on "white-box structural auditing" to catch such grafts.

Summary updated

Abstract

Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
BibTeX
@misc{sah2026cpu,
  author = {Tanmay Sah},
  title = {{A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights}},
  year = {2026},
  eprint = {2611.26450},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26450},
}
Figure