Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer

Yifan Zhang, Wei Bi, Kechi Zhang, Dongming Jin, Jie Fu, Zhi Jin

Mechanistic interpretability seeks to demystify the "black box" of neural networks by mapping internal weights to human-understandable concepts. However, standard Transformers present a significant hurdle due to superposition—the tendency to pack multiple features into overlapping, high-dimensional directions. This entanglement makes it exceptionally difficult to extract discrete, symbolic expressions from continuous weights.

While previous frameworks like MIPS have successfully extracted algorithms from Recurrent Neural Networks (RNNs), applying this paradigm to Transformers requires bridging the vast gap between dense continuous representations and sparse discrete logic. To address this, researchers introduce the Discrete Transformer, an architecture explicitly designed to be reverse-engineered into executable Python code, enabling de novo recovery of executable mechanisms directly from weights.

The Discrete Transformer Architecture

The Discrete Transformer relies on three architectural shifts to facilitate algorithm extraction:

  1. Functional Disentanglement: Inspired by the RASP (Restricted Access Sequence Processing) model, the architecture strictly separates routing and arithmetic. A Numerical Attention module is exclusively responsible for moving information, while a Numerical MLP handles mathematical operations.
  2. Numerical Residual Stream: Instead of dense vectors where additive updates cause interference, this model employs a collection of explicit scalar variables. By concatenating new outputs to the stream rather than adding them, the model preserves a clean "computational history."
  3. Temperature-Annealed Discrete Search: During training, the model uses a "soft" selection process to explore routing and arithmetic strategies. As training progresses, a temperature parameter is annealed, forcing the model to commit to "hard," discrete choices. This effectively crystallizes the model into a discrete computation graph.

Once the model is trained, the framework uses Hypothesis Testing to identify attention patterns (e.g., "attend to the previous token") and Symbolic Regression (PySR) to translate MLP operations into exact mathematical formulas.

Experimental Setup

The model was evaluated on an algorithmic reasoning benchmark adapted and expanded from the MIPS framework. The tasks span three categories:

  • Linear Arithmetic: Tasks such as adding the last two numbers (sum_last2) and physics-based simulations (gravity, spring).
  • Non-Linear Composition: Logical operations like XOR (parity_last2) and modular arithmetic (add_mod_3).
  • Continuous Dynamics: A novel addition of tasks involving floating-point variables, such as calculating exponential moving averages or free-fall heights, which previous integer-only autoencoders could not handle.

Key Findings and Performance

High Fidelity and Precision The extracted Python programs achieved perfect 1.00 accuracy on nearly all discrete tasks, matching the state-of-the-art MIPS baseline. Furthermore, the Root Mean Squared Error (RMSE) for these programs was exceptionally low (often between $10^{-6}$ and $10^{-9}$), indicating that the extracted code is a near-perfect replica of the model's internal mechanics.

Algebraic Equivalence The Discrete Transformer does not merely approximate logic; it discovers exact algebraic identities. For instance, it successfully recovered the precise mathematical formulation for XOR operations (e.g., $y = x_t + x_{t-1} - 2x_t x_{t-1}$) and piecewise-linear versions of max and min functions using ReLU units.

The "Crystallization" Phase By tracking "Discretization Discrepancy," researchers observed a two-stage learning process. The model initially learns a general, soft function before undergoing "structural crystallization," where it locks into a specific, discrete algorithmic path.

Inductive Bias as a Steering Wheel The architecture provides fine-grained control over synthesized programs. For example, if the MLP's mathematical capacity is restricted, the model adapts by solving the problem using only Attention—shifting from calculating a maximum to finding it within the context.

Limitations and Future Directions

The primary trade-off in the Discrete Transformer is between interpretability and expressivity. To ensure the model remains extractable, researchers had to limit the "fan-in" (the number of variables a single module can process simultaneously) and utilize a router-style attention mechanism.

While this approach is highly effective for algorithmic benchmarks and continuous physical dynamics, scaling it to the compositional complexity of Large Language Models (LLMs) remains an open challenge. Future work will aim to extend this framework to richer program classes and deeper computation graphs, further establishing the Discrete Transformer as a premier testbed for algorithm extraction and Transformer interpretability.

Summary updated

Abstract

Algorithm extraction aims to synthesize executable programs directly from models trained on algorithmic tasks, enabling *de novo* recovery of executable mechanisms from weights without relying on human-written target programs. However, applying this paradigm to Transformer is complicated by representation entanglement (e.g., superposition), where features encoded in overlapping directions substantially hinder the recovery of symbolic expressions. We propose the Discrete Transformer, an architecture explicitly designed to bridge the gap between continuous representations and discrete symbolic logic. By injecting discreteness through temperature-annealed sampling, our framework effectively leverages hypothesis testing and symbolic regression to extract human-readable programs. Empirically, the Discrete Transformer achieves performance comparable to the RNN-based MIPS baseline on shared discrete tasks, while broadening extraction to tasks with continuous-valued intermediate computations. Finally, we show that architectural inductive biases provide fine-grained control over synthesized programs, establishing the Discrete Transformer as a controllable testbed for algorithm extraction and Transformer interpretability.
BibTeX
@inproceedings{zhang2026weights,
  author = {Yifan Zhang and Wei Bi and Kechi Zhang and Dongming Jin and Jie Fu and Zhi Jin},
  title = {{Weights to Code: Extracting Interpretable Algorithms from the Discrete Transformer}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=nYzdAcLN5F},
}
Figure