ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models

Dhruv Dixit, Khushi Patel, Congcong Xu, Min SongarXiv 2026 · arXiv

Small language models (SLMs) are the "holy grail" for edge computing—they are cheap, fast, and private. However, when it comes to multi-step mathematical reasoning, they often fall flat. They tend to "hallucinate" arithmetic results or mimic the style of a solution without actually performing the underlying logic.

A new paper from researchers at the Stevens Institute of Technology introduces ARCHER (Action-trace Reinforcement for Calculator-grounded Hierarchical Executable Reasoning), a framework designed to turn SLMs into reliable, tool-using reasoning policies.

TL;DR

ARCHER moves small language models away from simply imitating text-based explanations toward learning an "executable policy." By training models to generate structured action traces and refining them with a novel hierarchical reinforcement learning objective, ARCHER significantly boosts mathematical accuracy across sub-10B models, outperforming strong tool-augmented baselines by an average of 5.5 percentage points.

What Problem Does This Paper Address?

The core issue is that standard "Chain-of-Thought" (CoT) training for SLMs is often just "rationale imitation." A model learns to write "Therefore, the answer is 42" because it looks like a correct solution, not because it actually calculated 42.

Existing tool-augmented methods (like giving a model a calculator) help, but they don't teach the model when to call the tool, how to format the call, or how to integrate the result back into its reasoning. ARCHER addresses this by treating reasoning as a sequence of discrete, auditable actions rather than just a string of text.

Methods

ARCHER is a two-stage framework:

  1. Stage 1: TRACE (Tool-augmented Reasoning via Action-Compiled Execution) The authors built a compiler that takes legacy CoT datasets (like GSM-8K) and converts them into structured "action traces." Instead of a paragraph of text, the model is trained on a sequence of typed actions: READ, PLAN, CALL_CALC, OBSERVE, VERIFY, and ANSWER. This stage uses Supervised Fine-Tuning (SFT) to teach the model the "language" of executable reasoning.

  2. Stage 2: HiGRA (Hierarchically Gated Reward Aggregation) After the model learns the basic structure, it is refined using Reinforcement Learning (specifically Group Relative Policy Optimization, or GRPO). The innovation here is the hierarchical reward system. Instead of a single "correct/incorrect" signal, ARCHER uses four reward families:

  3. Outcome Correctness: Did it get the right answer? (The dominant signal).

  4. Tool Grounding: Did it call the calculator when needed and use the result correctly?
  5. Process Discipline: Is the trace well-formatted and concise?
  6. Trace Alignment: Does the logic follow the "gold" reasoning path?

Crucially, these rewards are conditionally gated. For example, the model isn't rewarded for "alignment" if its answer is wrong, preventing it from learning to mimic the form of a correct answer while failing the logic.

Figure 1: Overview of ARCHER. Modules in Blue are trainable components, Light Gray are non-trainable or deterministic, Orange are Reward or evaluation, and Green are Data/traces/observations.

Datasets

The models were trained exclusively on the GSM-8K training set. Evaluation was performed across a wide battery of benchmarks to test generalization:

  • In-domain: GSM-8K.
  • Transfer: SVAMP, MMLU-Math, MATH, and MATH-500.
  • Competition-level: AIME-2024 and AIME-2025.
  • Multilingual: PolyMATH (covering 18 languages).

Key Insights and Ideas

  • Execution over Imitation: The biggest "aha" moment is that teaching a model to delegate arithmetic to a deterministic runtime (a calculator) is far more effective than trying to distill "math knowledge" into a small model's weights.
  • The Power of Gating: Flat reward mixtures in RL often lead to "reward hacking" where a model optimizes for a secondary metric (like formatting) while ignoring the primary goal (correctness). HiGRA’s priority-ordered gating ensures the model only gets "extra credit" for style if it actually solves the problem.
  • Complementary to "Thinking" Models: Interestingly, ARCHER provides gains even for models that already have "thinking" modes or have undergone reasoning-specific distillation (like DeepSeek-R1-Distill). This suggests that executable grounding is a fundamental improvement that stacks with other RL techniques.

Key Results

The researchers tested ARCHER across seven different backbones (including Qwen, Gemma, and Falcon).

  • Broad Gains: ARCHER improved macro-average accuracy by 5.51 percentage points over the strongest tool-augmented baseline (Chain of Abstraction).
  • Hard Tasks: On the difficult MATH-500 set, it saw a massive 8.95 point gain. On the competition-level AIME sets, it improved by over 5.3 points.
  • Training Dynamics: As shown in Figure 2, the TRACE stage provides a rapid initial boost in accuracy, while the HiGRA stage provides the steady refinement needed to reach peak performance.
Figure 2: Training dynamics study of ARCHER.

Limitations and Future Work

  • Tool Scope: Currently, ARCHER is focused on calculator use. The authors note that the action space could be expanded to include symbolic solvers (like SymPy), code execution, or retrieval tools.
  • Rule-Based Compilation: The TRACE compiler currently relies on rule-based parsing of existing datasets. Generalizing this to broader, less structured reasoning domains remains a challenge.
  • Language Sensitivity: While ARCHER improved PolyMATH scores, the authors noted that calculator grounding doesn't entirely remove the difficulties caused by language variation in multilingual settings.

In conclusion, ARCHER demonstrates that for small models to think big, they need to stop acting like writers and start acting like programmers. By grounding reasoning in executable actions, we can build compact models that are not only more accurate but also fully auditable.

Summary updated

Abstract

Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce \textbf{ARCHER}: \textbf{A}ction-trace \textbf{R}einforcement for \textbf{C}alculator-grounded \textbf{H}ierarchical \textbf{E}xecutable \textbf{R}easoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.
BibTeX
@misc{dixit2026archer,
  author = {Dhruv Dixit and Khushi Patel and Congcong Xu and Min Song},
  title = {{ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models}},
  year = {2026},
  eprint = {2611.18494},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18494},
}
Figure