WeightCode: Installing New Capabilities into Neural Networks via Weight-Space Programming

Bo Li, Taoyong Cui, Bob Zhang, Mengran Li, Chengyang Zhang, Yang Zhang, Pheng Ann Heng, Wanli Ouyang, Zhenfei Yin, Yingcheng Wu, Ling YangarXiv 2026 · arXiv

TL;DR

WeightCode introduces Weight-Space Programming (WSP), a paradigm where a frozen Large Language Model (LLM) writes executable code to directly modify the weights of a trained neural network. This allows researchers to "install" new capabilities—like compositional language rules or visual robustness—without gradient-based training, resulting in updates that are more data-efficient, interpretable, and reusable than traditional fine-tuning.

What Problem Does This Paper Address?

Neural networks are notoriously "brittle" when it comes to updates. If you want to add a new capability (e.g., teaching a robot a new action or making a vision model robust to fog), you typically have two choices: retraining from scratch or fine-tuning. Both are problematic. Fine-tuning often requires thousands of labeled examples and can lead to "catastrophic forgetting," where the model loses its original skills.

Furthermore, we often have explicit task knowledge—like a rule ("if you see X, do Y twice") or a few demonstrations—that is hard to "inject" into a sea of millions of floating-point weights. WeightCode asks: Can we treat a neural network's weights as a programmable address space? Instead of using gradients to "nudge" weights, can we just write a program to "patch" them?

Methods

The authors propose WeightCode, a framework for Weight-Space Programming. The core idea is a loop involving a frozen LLM (like Claude 3.5 Opus) acting as a programmer:

  1. Inspection: The LLM reads the target network's weights ($W$) and task knowledge ($I$), such as a grammar rule or a few demonstrations.
  2. Programming: The LLM writes an executable Python program $p$. This program isn't a new model; it's a script that calculates a weight edit: $W' = p(W, I)$.
  3. Verification: The program is run in a sandbox. It might perform SVD, analyze activation statistics, or find "idle" neurons to repurpose.
  4. Feedback: The edit is tested against a small "check" set. If it improves the new task without breaking old ones, it's kept.

Crucially, the resulting "weight program" is reusable. Because it's code that looks at network statistics to decide how to change weights, you can run the same program on a different, independently trained network of the same architecture, and it will compute a custom, effective patch for that specific model.

Overview of the WeightCode framework showing how an LLM writes a program to update a network.

Datasets

The authors evaluated WeightCode across three diverse domains:

  • Language (SCAN): A benchmark for compositional generalization. The goal is to teach the model to handle new command combinations (e.g., "jump twice") based on a grammar rule.
  • Navigation (BabyAI): A grid-world environment where the model must learn a new "pickup" action from just 8 demonstrations.
  • Vision (ImageNet & CIFAR-10): Improving robustness to common corruptions (Gaussian noise, contrast reduction) using the ImageNet-C and CIFAR-10-C benchmarks.

Key Insights and Ideas

  • Repurposing "Dark Matter": The LLM often finds "dead" or underutilized neurons (e.g., in MLP blocks) and rewrites them to act as gates. For example, in a Vision Transformer, a program might turn an idle neuron into a "noise-correction gate" that activates only when it detects a specific noise signature, then subtracts that noise from the internal representation.
  • Code is More Portable than Deltas: Simply copying weight changes ($\Delta W$) from one model to another usually fails because different models represent features in different "coordinates." However, a program that says "find the direction of maximum noise and subtract it" works across models.
  • Succession and Stacking: Capabilities can be stacked. You can install a noise-robustness patch, and then a second program can install a contrast-robustness patch on top of it without breaking the first.
Causal intervention showing how installed gates drive robustness gains.

Key Results

  • Extreme Data Efficiency: On the SCAN task, WeightCode achieved a 99.8% exact match on new commands by changing only 257 weights, whereas fine-tuning on the same data achieved nearly 0%.
  • Outperforming Fine-Tuning: On ImageNet, weight programs achieved net gains of +10.47 to +17.62 points in robustness, outperforming the best of eight different fine-tuning methods (like LoRA) under the same data constraints.
  • Reliable Reuse: When a program written for one Vision Transformer was rerun on five other independently trained ones, it retained 83–110% of its original gain. In contrast, copying the raw weight changes resulted in zero gain.
Comparison of rerunning a program vs. copying weight edits across different networks.

Limitations and Future Work

  • LLM Dependency: The method relies on high-reasoning LLMs to understand network architectures and write the code. It is currently limited to architectures the LLM can "reason" about.
  • Search Space: While the LLM is efficient, it still requires a few "checks" (trials) to find the best program. Scaling this to massive, multi-step capabilities remains a challenge.
  • Scope of Edits: The authors limited edits to 20% of the network's weights. It's unclear if WSP can handle fundamental architectural shifts or entirely new modalities.
  • Future Direction: The authors suggest building "reusable program libraries" that can be shared to patch models across the community, much like software libraries.

Summary updated

Abstract

Neural networks encode learned capabilities in their weights, yet extending those capabilities with explicit task knowledge remains a fundamental challenge. Here we show that Weight-Space Programming (WSP), in which executable programs read, construct or modify a network's weights directly, can install new capabilities within trained neural networks. WeightCode, a WSP framework, uses a frozen language model to write executable weight programs that install new computations into trained neural networks using task knowledge and their existing weights, without gradient-based training. Once installed, the new capabilities run independently of the language model. Given a grammar or eight demonstrations, weight programs enable previously unseen command compositions (0% to 99.8% exact match) and a new pickup action (0% to 98.4% success), while preserving the original command and navigation capabilities. On ImageNet, four weight programs achieve net gains of 8.84-17.62 points while preserving clean accuracy, outperforming the best of eight fine-tuning methods under matched examples, checks, and retention constraints. In a Vision Transformer, causal interventions show that installed noise-correction gates drive robustness gains, while their selective activation preserves clean accuracy. Rerun on five independently trained Vision Transformers without further language model calls, ten weight programs retain 83-110% of their source-network gains on average, whereas copying their fixed weight edits yields negligible gains. Six programs written for an already edited network add 5.23-14.50 points of contrast robustness while retaining the previously installed noise gain. Together, these results establish WSP as a new route to acquiring and accumulating capabilities in trained neural networks through reusable weight programs.
BibTeX
@misc{li2026weightcode,
  author = {Bo Li and Taoyong Cui and Bob Zhang and Mengran Li and Chengyang Zhang and Yang Zhang and Pheng Ann Heng and Wanli Ouyang and Zhenfei Yin and Yingcheng Wu and Ling Yang},
  title = {{WeightCode: Installing New Capabilities into Neural Networks via Weight-Space Programming}},
  year = {2026},
  eprint = {2611.25476},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25476},
}
Figure