CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse

Peter Belcak, Greg Heinrich, Saurav Muralidharan, Shizhe Diao, Pavlo MolchanovarXiv 2026 · arXiv

TL;DR

CPU-ARC introduces "Agentic Reasoning Caching" (ARC), a novel framework that converts expensive Large Language Model (LLM) reasoning traces into efficient, CPU-executable Python programs. By decoupling the discovery of a solution from its execution, it achieves frontier-level performance on complex tasks while being up to 291x cheaper and fully explainable.

What Problem Does This Paper Address?

Large Language Models are notoriously inefficient: they "reinvent the wheel" for every query, solving common problems from scratch even when they have encountered identical logic before. Existing system-level caching (SLC) methods fall into two camps, both with flaws. "Monolithic" caches are brittle, failing if a user changes a single word or number. "Modular" caches decompose tasks but still require significant GPU compute to re-generate answers.

The authors identify a massive opportunity in the fact that many problems are equirational—they share the same underlying reasoning structure. As illustrated in the paper's introductory example, the logical steps to find the divisors of 196 are identical to those for 144; only the arithmetic changes. CPU-ARC aims to capture this reusable logic in a way that is robust to rephrasing and reparameterization.

Figure 1: An example of equirational problems where the underlying reasoning remains identical despite changes in phrasing or numerical parameters.

Methods

The core of the proposal is the Agentic Reasoning Cache (ARC). Instead of caching text, it caches Semantic Functions—executable Python abstractions of reasoning steps.

Key technical components include:

  • Semantic Discovery: Unlike standard code, these functions reference inputs and other functions using natural language descriptions (e.g., load(ctx, "the numeric temperature value")). This allows the system to "find" the right data or sub-routine even if the variable names aren't hardcoded.
  • The Builder ($\mathcal{B}$): An expensive "builder" model (like Claude Opus 4.5) analyzes a reasoning trace and decomposes it into a network of these semantic functions.
  • The Target ($\mathcal{T}$): At inference time, the system uses a tiny, cost-efficient model (e.g., a fine-tuned 1B-parameter Llama 3.2) to handle basic NLP tasks like value extraction, while the heavy lifting is done by deterministic Python code on a CPU.
  • Compilation: Once a cache is stable, it can be "compiled" into a standalone Python module, replacing dynamic semantic lookups with direct function calls for near-zero latency.

Datasets

The authors evaluated CPU-ARC on a battery of difficult mathematical and scientific benchmarks to prove it can handle "frontier-level" intelligence:

  • MATH-500 & AIME (2024-2026): High-school and undergraduate competition-level mathematics.
  • GPQA Diamond: Graduate-level science questions (physics, chemistry, biology) designed to be "Google-proof."
  • GSM8K: Grade-school math word problems used to test basic reasoning chains.
  • MMLU-Pro & SciBench: Multidisciplinary knowledge and university-level textbook science problems.

Key Insights and Ideas

  • Decoupling Search from Execution: The most profound insight is that finding a solution is hard (requiring a massive LLM), but executing that solution is often just a series of logical steps and arithmetic that a CPU can handle.
  • Explainability as a Side Effect: Because the cache is literally code, it is fully transparent. The system uses a narrate() primitive to record a human-readable trace of its "thought process," making it far more auditable than a standard LLM.
  • Deterministic Superiority: In several cases, the cache outperformed the source LLM. Why? Because once the logic is captured in code, the system stops making the "silly" arithmetic errors that plague even the best language models.
  • Small Models are Enough: The paper demonstrates that a 1B-parameter model is sufficient to drive these "fractional agents," suggesting that complex reasoning can be moved from massive cloud GPUs to local edge devices.

Key Results

  • Accuracy: Cache accuracy matched or exceeded the source model on 7 out of 8 benchmarks. For instance, on AIME '25, the cache achieved 93.3% accuracy compared to the source model's 83.3%.
  • Cost Savings: The "Target cost" (invoking the cache) was 16x to 291x cheaper than the "Source cost" (calling the frontier LLM).
  • Robustness: The system maintained high performance (>90% pass rate) across all perturbation modes, meaning it successfully generalized beyond the specific wording or numbers of the original training examples.
  • Efficiency: The one-time construction cost is amortized quickly; for AIME problems, the cache "pays for itself" after just 13 similar queries.

Limitations and Future Work

The authors acknowledge that they haven't yet addressed the "merging" problem: how to automatically combine multiple similar semantic functions into a single, more generalized one.

Looking forward, they suggest that this approach could lead to a new class of "Semantic Models"—a layer of explainable, efficient software that sits on top of neural models. They call this vision "neuralware," where LLMs act as a support layer for intelligent, deterministic software rather than being the entire stack themselves.

Summary updated

Abstract

Large language models reinvent the wheel all the time. They conflate task-solving with the execution of the solutions, which forces them to solve every problem from scratch. We show that once a task has been reasoned through, the application of its solution to a new problem instance can be done very efficiently and with little involvement from language models. We introduce agentic reasoning caching, a method for constructing robust CPU-executable agentic reasoning caches (ARCs) from LLM reasoning traces. ARCs show on-par performance with frontier LLMs across a range of difficult mathematical and scientific tasks, are fully explainable, continually trainable, robust to input perturbations, and 16-291x cheaper to invoke than the source model.
BibTeX
@misc{belcak2026cpu,
  author = {Peter Belcak and Greg Heinrich and Saurav Muralidharan and Shizhe Diao and Pavlo Molchanov},
  title = {{CPU-ARC: Agentic Reasoning Caching for on-CPU Reasoning Reuse}},
  year = {2026},
  eprint = {2611.09891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09891},
}
Figure