Weight-Space Programming Doesn't Need Transformers

Hyunjong Ok, Jaeho LeearXiv 2026 · arXiv

TL;DR

When Large Language Models (LLMs) are tasked with directly writing the weights of a model—a paradigm called Weight-Space Programming (WSP)—they don't build Transformers. Instead, they naturally converge on sparse associative memories that are up to 680× smaller and 17× faster than Transformers, while being significantly easier to debug and edit.

What Problem Does This Paper Address?

Standard neural architectures like the Transformer were born from the "hardware lottery": they are exceptionally well-suited for gradient descent and run fast on GPUs. However, a new paradigm is emerging where frontier LLMs can directly "write" the weights of a target model without any gradient-based training.

The authors ask a fundamental question: Should weight-space programming keep the Transformer? In a world where weights are authored line-by-line by an AI rather than optimized by calculus, the dense, overlapping representations of Transformers might actually be a liability. The paper investigates whether there is a more "writable" and efficient architecture for this new way of building AI.

Methods

The researchers evaluated three frontier LLMs—Claude Opus 5.5, GPT-6.1 Sol, and GPT-6 Astra—as "writers." These writers were given a task (like answering science questions), ten labeled examples, and an execution sandbox.

The study compared two regimes:

  1. Fixed Architectures: The LLM writes weights for a provided architecture (e.g., a Transformer or RNN).
  2. Free Design: The LLM is free to design the architecture itself.

Observing that all three LLMs independently converged on a "sparse fact table" design, the authors proposed a formal associative-memory-based model. This model uses a concept table (first-order memory) and a pair table (second-order memory) to store facts. Unlike Transformers that use "soft" attention over dense vectors, this model uses "hard" top-$k$ selection over sparse keys, making every weight entry traceable to a specific plain-text fact.

Comparison of the standard Transformer architecture versus the LLM-designed sparse table models.

Datasets

The primary evaluation used the ARC (AI2 Reasoning Challenge), consisting of 3,530 multiple-choice science questions. To test generalization, the authors also ran experiments on:

  • SciQ, OpenBookQA, MMLU, and MathBench: For science and math reasoning.
  • SQuAD 1.1: For text generation and span extraction.
  • CIFAR-10: For image classification.
  • AG News: For text classification.

Key Insights and Ideas

  • The Writable Architecture: For an LLM to write weights effectively, it needs two properties: Traceable answers (knowing exactly which weight caused a mistake) and Local edits (changing one weight shouldn't break unrelated predictions). Transformers fail here because their dense weights "blur" facts together.
  • Convergence of Design: Despite no specific prompting, all three frontier LLMs abandoned Transformers in favor of sparse retrieval. They preferred compiling plain-text facts into rows of a table.
  • Interference is the Enemy: When LLMs try to write facts into a Transformer, the overlapping dense keys cause "interference," leading to lower accuracy than simple keyword search (BM25). Sparse tables eliminate this overlap.
  • Associative Memory as a Unified Framework: The paper elegantly frames both Transformers and their new model as associative memories. The difference lies in the selection function: Transformers use a "soft" softmax, while WSP-optimized models use a "hard" max or top-$k$ selection.
The efficiency and speed advantages of the proposed associative-memory model compared to Transformers.

Key Results

  • Efficiency: LLM-designed models achieved higher accuracy on ARC than the best Transformer the same LLM could write, while being ~1,800× smaller in terms of parameters.
  • Speed: The authors' associative-memory-based model is up to 17× faster on a GPU than a same-sized Transformer because it avoids the heavy matrix multiplications of dense layers.
  • Accuracy: On the ARC benchmark, the best LLM-designed lineage reached 54.4% test accuracy, outperforming gradient descent trained from scratch on the same small data.
  • Editability: In the table-based models, removing a single written line (fact) changed only about 0.5 to 1.8 answers on average. In contrast, removing a line in a written Transformer could disrupt hundreds of answers due to weight interference.
An illustrative example of how a question about bird adaptation traces directly to a specific written fact in the table.

Limitations and Future Work

  • The Pretraining Gap: While WSP beats training from scratch on small datasets, it still trails behind massive pretrained models (like Qwen3.5-0.8B at 82.1% accuracy). WSP is currently a way to build specialized, interpretable models, not a replacement for foundation model pretraining.
  • Reasoning vs. Recall: Current WSP models primarily function as sophisticated fact-retrieval engines. They don't yet "chain" facts for multi-step reasoning or compose complex long-form answers.
  • The Vision Hurdle: WSP struggles with raw pixels because individual pixel weights don't have "nameable" meanings that an LLM can easily reason about. It works much better when using high-level tokens from a pretrained encoder like DINOv2.
  • Future Directions: The authors suggest moving toward WSP that can build its own internal representations and co-designing hardware specifically for the sparse lookups these models favor.

Summary updated

Abstract

Standard neural architectures such as the Transformer were chosen because they train well with gradient descent and run fast on GPUs. In weight-space programming, a large language model (LLM) writes target weights by hand without gradient-based training, so those reasons no longer apply. Should we keep the Transformer? Given minimal guidance, ten labeled examples, and a sandbox, mainly on multiple-choice questions, we task three frontier LLMs with either generating the weights of a fixed architecture or designing the architecture themselves. When free to design, all three compile plain-text facts into a sparse table queried by a fixed procedure, reaching up to 54.4% test accuracy. An LLM-designed model achieves higher test accuracy than the most accurate released Transformer that the same LLM wrote while using about 1,800x fewer parameters. Without gradients, they choose a table, in which the LLM can state what each weight means and can find and rewrite a wrong one. Each wrong answer points to one row, and removing a row changes about one other answer. From this, we propose an associative-memory-based model up to 17x faster on GPUs than a same-sized Transformer. We also test pretrained weights, other benchmarks, and tasks where models generate answers. Pretrained LLMs remain more accurate; the claim concerns which structure to build when writing weights.
BibTeX
@misc{ok2026weight,
  author = {Hyunjong Ok and Jaeho Lee},
  title = {{Weight-Space Programming Doesn't Need Transformers}},
  year = {2026},
  eprint = {2611.22055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22055},
}
Figure