Replicating and Extending Weight-Space Programming: Frontier LLMs Compile Programs into Weights and Read Input Layers

Roksana Goworek, Alexander Hickey, Federico Nanni, Martin O'ReillyarXiv 2026 · arXiv

TL;DR

Frontier LLMs like GPT-6 Astra and Claude Opus 5.5 have developed the startling ability to "write" working neural networks from scratch and "read" raw weights to explain their function. This paper reveals that this isn't magic or "intuition"—the models are essentially compiling symbolic programs into sparse, high-magnitude weight matrices that act like digital logic gates.

What Problem Does This Paper Address?

Neural networks have traditionally been "grown" through gradient descent, resulting in "black box" weights that are notoriously difficult for humans to interpret. Recently, researchers at MIT claimed that the latest frontier model, GPT-6 Astra, could bypass training entirely, writing the weights for small game-playing networks directly in a single text response.

The authors of this paper, a team from The Alan Turing Institute, set out to verify these claims. They wanted to know: Is this just memorization of training data? How are the models actually doing it? And can this ability be used for "weight-space auditing"—detecting backdoors or malicious behavior in a model just by looking at its numbers?

Methods

The researchers conducted a high-intensity "three-day sprint" to replicate and extend the MIT results using several methods:

  • Replication & Generalization: They tested if Astra could write weights for games like Breakout, including "unseen" versions where the board was upside down or controls were swapped, to rule out simple memorization.
  • Code-before-Weights: They compared the models' ability to write a strategy in Python versus writing it in weights to see which capability emerged first.
  • Decompilation: They asked LLMs to perform "reverse engineering"—taking the raw weights of a network and translating them back into human-readable Python code.
  • Weight-Space Auditing: They presented models with the plaintext weights of small language models (Olmo 3 190M) that had been "poisoned" to produce gibberish when triggered, asking the LLM to identify the compromised model.

Datasets

The study utilized several benchmarks and custom environments:

  • MinAtar: A simplified version of the Atari benchmark (Breakout, Seaquest, Asterix, Freeway, Pong).
  • MNIST: The classic dataset of 70,000 handwritten digits.
  • Custom Games: Three new games designed by the authors (Reverse Space Invaders, Whac-A-Mole, and a Shooter) to test "native" weight reading on weights the models couldn't have seen.
  • Olmo 3 190M & Qwen3.5-0.8B: Used for the auditing tasks, involving both poisoned base models and malicious adapters.

Key Insights and Ideas

  • Compilation, Not Intuition: The models aren't "feeling" their way to a solution. They design a symbolic, rule-based strategy first and then "compile" it into weights. These networks are extremely sparse (often <10% non-zero weights) and use massive weight magnitudes (up to 1,000) to act as on/off switches.
  • The Input Layer is the Map: When LLMs "read" weights, they are primarily looking at the first layer. Because each input channel corresponds to a specific game object (ball, paddle, wall), the weight norms form a spatial heatmap. If you shuffle the weights, the LLM's "understanding" vanishes.
  • The "Fuzzy" Failure: While Astra can write a perfect Breakout player (a symbolic task), it struggles with MNIST (a "fuzzy" pattern recognition task). It scored only ~50% on digits, failing because there is no simple "if-then" program to compile for computer vision.
  • Evolutionary Leap: This capability is brand new. Models from late 2025 (like Claude Opus 4.5) could barely write the Python code for these games, let alone the weights. By late 2026, the "compilation" ability had fully emerged.

Key Results

  • Breakout Performance: Astra’s hand-written weights scored up to 550 in Breakout, matching the MIT results and far exceeding a standard PPO-trained baseline (score of 25).
  • Perfect Decompilation: In a stunning result, LLMs were able to decompile Astra’s weights back into Python code that matched the network's actions in 100% of tested game states.
  • Auditing Success: Astra correctly identified a poisoned language model in 10 out of 10 runs by spotting "freshly initialized" embeddings for rare tokens.
  • Auditing Limits: While the models could find that a model was poisoned, they never successfully identified the specific trigger phrase or described the malicious behavior accurately.

Limitations and Future Work

The authors are careful to temper the hype. Currently, this only works for tiny networks (thousands of weights, not billions). The "reading" ability is also "shallow"—it’s great at identifying spatial layouts but poor at understanding complex logic or non-spatial mechanics.

Future research should focus on:

  1. Tool-Assisted Auditing: Allowing LLMs to use Python scripts to analyze weights rather than reading them as raw text.
  2. Interpretability Benchmarks: Creating standardized tests to track how LLMs improve at mechanistic interpretability.
  3. Scaling: Investigating if these "compilation" techniques can be applied to larger, more complex architectures.

Summary updated

Abstract

Researchers at MIT recently reported that GPT-6 Astra, a frontier large language model (LLM), can write the weights of small neural networks directly in a single reply, and can read the raw weights of small networks to say what they were trained to do. In a three-day sprint, we replicated and extended these results. Our four findings: (1) Writing weights reproduces. Astra's networks for the game Breakout scored 514, 550 and 307 (versus MIT's 517), and scored just as well on two modified versions of Breakout. Claude Opus 5.5 can also write working networks, but often runs out of output tokens. (2) What is new is translating programs into weights. Earlier models could write a good policy as code a generation before they could write network weights, and the networks the models write behave like compiled programs: two models independently decompiled Astra's Breakout network into Python matching its actions in all tested states. On classifying handwritten digits, Astra's networks scored 50% on average, below a simple classifier (82%). (3) Reading weights is mostly reading the input layer. Astra and Opus 5.5 identify which game a network plays almost perfectly, but on three new games we designed and trained ourselves, Astra recovered where things are on the board, while mostly missing how the game works. (4) Auditing from weights alone is possible but shallow. Without executing code, Astra picked out a small language model poisoned during training in 10 of 10 runs, one poisoned by extra training afterwards in 7 of 10, and a malicious adapter in 23 of 32 cases, but never identified the trigger or malicious behaviour. Open-weight models lag behind. From this early study, we conclude that LLMs can read and write small networks, but that this currently amounts to compiling hand-designed programs and reading the layout of the input layer. It is a promising but early tool for interpretability.
BibTeX
@misc{goworek2026replicating,
  author = {Roksana Goworek and Alexander Hickey and Federico Nanni and Martin O'Reilly},
  title = {{Replicating and Extending Weight-Space Programming: Frontier LLMs Compile Programs into Weights and Read Input Layers}},
  year = {2026},
  eprint = {2611.21449},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21449},
}
Figure