A Tool in the Forward Pass: Frozen Tape Modules as Layers of a Small Language Model

Michael Granitzer, Saber Zerhoudi, Mohammad Al-Maamari, Hai Dang Dang, Kanishka Ghosh DastidararXiv 2026 · arXiv

TL;DR

Researchers have developed "tape layers"—small, frozen neural modules that perform specific computations directly within a language model's forward pass. By bypassing the need to generate text for tool calls, these modules can solve complex reasoning tasks like object tracking 10x faster than traditional tool-use methods and over 600x faster than Chain-of-Thought reasoning, all while maintaining near-perfect accuracy.

What Problem Does This Paper Address?

Large Language Models (LLMs) often struggle with precise multi-step reasoning, such as arithmetic or tracking objects through a series of swaps. The standard solution is "tool use": the model generates a text-based command (e.g., a Python snippet), an external interpreter runs it, and the result is fed back into the model.

However, this approach is surprisingly expensive. Every token the model generates requires a full forward pass through billions of parameters. For a simple calculation, the "overhead" of writing out the tool call can cost far more than the computation itself. This paper explores a more efficient alternative: can we embed the tool's logic inside the model's existing layers so the computation happens "for free" during the initial prompt processing?

Methods

The authors introduce the Tape Layer, a three-part architecture integrated into a frozen LLM (specifically tested on Gemma-3-1b-it):

  1. The Writer: A small, learned network that reads the model's hidden states at an early layer. It identifies relevant tokens (like numbers or names) and "writes" them onto a symbolic tape—a short sequence of discrete symbols.
  2. The Tape Module: This is the "brain" of the tool. It's a tiny neural network (only 64k parameters) trained independently to perform a specific task by iteratively rewriting the symbolic tape until only the result remains. Crucially, once trained, this module is frozen.
  3. The Reader: Another small learned network that takes the final symbol from the tape and injects its meaning back into the model's hidden states, allowing the upper layers of the LLM to "see" the answer.
Figure 1: One computation, three interfaces. (a) The model generates a program for an interpreter. (b) It generates only the inputs to a fixed tool. (c) A learned writer constructs those inputs directly from hidden states. A frozen neural module computes the result, and a reader returns it to the model. The tape traces Bob’s final object back through the swaps to its original owner. (d) The layer’s components. (e) Measured GPU kernel time at seven people, relative to the layer, with each method’s accuracy (§5.2).

Datasets

The researchers evaluated their approach on two primary tasks:

  • Arithmetic: Evaluating expressions (modulo 10) with varying numbers of operators.
  • Object Tracking: The "tracking shuffled objects" task from the BIG-Bench Hard (BBH) benchmark. This involves following items as they are swapped between people (e.g., "Alice swaps with Bob...").
  • General Benchmarks: They used MMLU and snippets from The Pile to ensure that adding these specialized layers didn't "break" the model's general knowledge or text generation capabilities.

Key Insights and Ideas

  • Decoding is the Bottleneck: The primary "aha moment" is that the cost of tool use isn't the computation—it's the decoding. By moving the tool into the forward pass, you eliminate the need for sequential token generation.
  • Length Extrapolation: Because the tape module learns local rewriting rules (e.g., "if X swaps with Y, and I'm looking for X, now look for Y"), it can handle much longer sequences than it saw during training. A module trained on 5 swaps can perfectly handle 7 or more because it just runs for more "rounds" on the tape.
  • Modular Composition: You can plug multiple independent tape layers (one for math, one for tracking) into the same model. By using a "learned gate," the model only activates the relevant layer when it sees a matching prompt, keeping the layers "silent" during normal conversation.

Key Results

The performance gains are dramatic, particularly in terms of efficiency:

  • Speedup: On the object tracking task, the tape layer required only 1/10th of the GPU kernel time compared to a text-tool baseline. Compared to zero-shot Chain-of-Thought (CoT), the tape layer was 621x faster.
  • Accuracy: The tape layer achieved 1.00 accuracy on the BBH object tracking task with seven people, outperforming both the text-tool baseline (0.99) and CoT (0.93) on a 4B parameter model.
  • Zero Interference: When tested on the MMLU benchmark, a model equipped with two tape layers performed identically to the base model, proving that these "internal tools" can be added safely.
Comparison of accuracy and GPU time. The tape layer (top) achieves perfect accuracy at a fraction of the computational cost of other methods.

Limitations and Future Work

While promising, the authors note several hurdles for broader adoption:

  • Input Extraction: While the "tape module" is very good at computing, the "writer" still struggles to reliably extract inputs from highly unstructured or "messy" text it hasn't seen before.
  • Single-Symbol Output: Currently, the system is designed for tools that return a single discrete symbol (like a digit or a name). Extending this to multi-token or continuous outputs is a significant next step.
  • Training Stability: Learning the "writer" from final answers alone (without labeled tapes) is possible but remains less reliable than supervised training.

The authors conclude that while learned modules are interchangeable with exact code for these tasks, the real value lies in the interface: accessing bounded computations through hidden states is a powerful way to make small models punch far above their weight class without the latency of text generation.

Summary updated

Abstract

Calling a tool through generated text adds decoding steps, even when the computation itself is short. We study an alternative: a small neural module that runs inside a language model's forward pass. A learned writer turns hidden states into a symbolic tape, a short sequence of discrete symbols that encodes the computation. The module, trained separately and then frozen, reduces the tape to a result, and a learned reader returns that result to the model. We evaluate arithmetic and object tracking in gemma-3-1b-it. With a writer trained on labelled tapes, the layer answers all 750 items of BIG-Bench Hard's tracking shuffled objects correctly. At seven people it needs about 1/10 of the GPU kernel time of a text-tool baseline that uses the same host model and exact code, with accuracy 1.00 against 0.99. The writer can also be learned from answers alone, guided by a hand-written copying prior, but less reliably: 5 of 6 runs succeed in the 1B host and 2 of 4 in the 4B host. Two independently trained layers keep their task accuracy when attached to one model, and a learned gate keeps them silent on the tested general-text and look-alike prompts. The results support a narrow use case: a bounded computation that returns one symbol can avoid the decoding cost of a textual tool call. Extracting the right input reliably, and deciding when to act, remain the main open problems.
BibTeX
@misc{granitzer2026tool,
  author = {Michael Granitzer and Saber Zerhoudi and Mohammad Al-Maamari and Hai Dang Dang and Kanishka Ghosh Dastidar},
  title = {{A Tool in the Forward Pass: Frozen Tape Modules as Layers of a Small Language Model}},
  year = {2026},
  eprint = {2611.00783},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00783},
}
Figure