WeightBench: Evaluating Language Models as Weight-Space Programmers

Aayush Marasini, Barsat Khadka, Nick RahimiarXiv 2026 · arXiv

TL;DR

WeightBench is a new benchmark designed to evaluate Large Language Models (LLMs) as "Weight-Space Programmers"—agents that can directly read, write, and repair neural network parameters without using gradient descent. The results show that while frontier models like GPT-6.1-sol and Claude 5.5 Opus can synthesize simple control policies from scratch, they still struggle with surgical edits to large models and identifying impossible tasks.

What Problem Does This Paper Address?

Traditionally, neural networks are "black boxes" created through optimization (gradient descent). We don't usually "program" their weights; we train them. However, recent research suggests that LLMs might be able to treat weights as structured data—reading a network's purpose from its parameters or fixing a bug by manually adjusting a few floating-point numbers. This capability is called Weight-Space Programming (WSP).

The challenge is that we lack a systematic way to measure WSP. Existing benchmarks don't account for the unique constraints of weight manipulation. For example, how do we ensure an LLM is actually reasoning about the weights rather than just running a hidden "trial-and-error" optimization loop? How do we measure if a "fix" to one behavior accidentally destroys the rest of the model's capabilities? WeightBench aims to provide a rigorous, audited environment to answer these questions.

Methods

The authors introduce a framework that treats LLMs as agents operating within a strictly controlled environment. Key components include:

  • The WeightBench Benchmark: 42 tasks across five tracks: READ (diagnosing model behavior), FIX (surgical patching of bugs), META (identifying impossible edits), WRITE (synthesizing weights from scratch), and SYSTEMS (hardware-aware optimization).
  • Strict Auditing & Sandboxing: To prevent agents from "cheating" via gradient descent, they are placed in an OS-level sandbox. The authors use a metered NumPy subclass that traces arithmetic operations at the operator level to detect and block hidden forward-pass chains or optimization loops.
  • Composite Scoring ($S_{WB}$): Success isn't just about solving the task. The score factors in:
  • Task Gain ($S^+$): Did it solve the problem?
  • Collateral Damage ($D_{col}$): Did it break unrelated capabilities (e.g., MMLU score)?
  • Structural Edit Cost ($u_{edit}$): How "surgical" was the change?
  • Compute Use: How many resources were expended?
Figure 1. The WeightBench evaluation pipeline.

Datasets

WeightBench evaluates agents on a diverse set of "subject models" ranging from tiny 2K-parameter MLPs (control policies for games like Breakout) to 7B-parameter Transformers.

To prevent models from simply memorizing weight indices from public datasets, the authors use Anti-Memorization techniques. They procedurally regenerate subject models using fresh random seeds and apply seeded orthogonal rotations and hidden-dimension permutations. This scrambles the internal geometry while preserving the functional behavior, forcing the LLM to actually analyze the weights.

Key Insights and Ideas

  • Zero-Shot Synthesis is Possible: One of the most striking findings is that top-tier models can perform "Zero-Shot Weight Synthesis." They can take a blank slate of zeros and write the exact weight values needed to create a working game-playing policy without ever running a training loop.
  • The "Surgical" Precision Gap: While models are decent at reading weights, they are much worse at fixing them. Most agents failed to perform "surgical" fixes on large language models without causing significant collateral damage to the model's general intelligence.
  • Lack of Meta-Awareness: Current LLMs struggle to distinguish between a very difficult task and a mathematically impossible one. In the META track, models often tried to force an edit on an impossible task rather than refusing it.
  • Scaling Matters: There is a clear performance hierarchy. Larger, more capable models (like Claude Opus vs. Haiku) show significantly better weight-space reasoning, suggesting WSP is an emergent capability of scale.

Key Results

The benchmark results (summarized in the table below) show that GPT-6.1-sol and Claude 5.5 Opus are the current leaders in weight-space programming.

Agent performance across the five WeightBench tracks. GPT-6.1-sol and Claude 5.5 Opus lead the pack, particularly in the WRITE track.
  • Top Performance: GPT-6.1-sol achieved a macro score of 0.568 on the 38 tasks it was tested on, followed closely by Claude 5.5 Opus at 0.535 (on all 42 tasks).
  • Track Variance: Models performed best on TR-01 (READ) and TR-04 (WRITE). However, performance plummeted on TR-05 (SYSTEMS), where agents struggled with data-free compression and hardware optimization.
  • DeepSeek Struggles: DeepSeek v4-flash had the lowest overall score (0.215) and the highest "quarantine" rate, often triggering the audit guards by attempting prohibited operations.

Limitations and Future Work

The authors acknowledge that WSP is still in its "infant phase." Current models lack the precision required for reliable, industrial-grade model editing.

Future directions include: * Expanding the benchmark to include even more capable models like Claude Fable 5.1. * Exploring "dynamic difficulty scaling" to keep the benchmark challenging as LLMs continue to improve. * Refining the "Surgical Patching" tasks to better understand why models struggle to preserve general capabilities while fixing specific bugs.

Summary updated

Abstract

Recent work shows that large language models (LLMs) can operate directly on the parameters of other neural networks. They can read what a network was trained to do, fix a network that fails, and write working networks from scratch, all without gradient descent. This capability has been termed Weight-Space Programming (WSP). Exploring this new paradigm systematically requires a robust benchmark. We introduce WEIGHTBENCH, a benchmark that evaluates LLMs of increasing capability on 42 WSP tasks across five tracks: reading, fixing and writing weights, recognizing edits that are impossible, and data-free compression. The networks whose weights the LLMs read and edit range from 2K-parameter control policies to 7B-parameter language models. Experimental benchmark results show that frontier agents can already read small networks and write working control policies without ever running them, but cannot yet patch language models, compress them without data, or tell an impossible edit from a hard one. On all 42 tasks, Claude Opus 5.5 reaches a macro score of 0.535 (Sonnet 5.5 0.466, Haiku 5.5 0.402, DeepSeek v4-flash 0.215); on the 38 tasks it was run on, GPT-6.1-sol reaches 0.568 against Opus's 0.546. Code, the task registry, our OS-level sandbox harness, and the live leaderboard are publicly available at https://github.com/Aayush-Marasini/WeightBench.
BibTeX
@misc{marasini2026weightbench,
  author = {Aayush Marasini and Barsat Khadka and Nick Rahimi},
  title = {{WeightBench: Evaluating Language Models as Weight-Space Programmers}},
  year = {2026},
  eprint = {2611.18553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18553},
}
Figure