Training Vision Models for LLM-Driven Weight-Space Programming

Timo Ropinski, Sebastian KranzarXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) can now act as autonomous "weight-space programmers," fixing neural network errors by directly modifying their weights without any gradient descent. This paper introduces Edit-Ready Training (ERT), a novel training method that "rehearses" these manual edits during the initial training phase, making vision models significantly more cooperative and easier for LLMs like GPT-6 and Claude to repair.

What Problem Does This Paper Address?

When a deployed image classifier consistently misidentifies a specific subgroup (e.g., mistaking a "red-winged blackbird" for a generic "waterbird") or confuses two similar classes, the traditional fix is to collect more data and fine-tune the model. However, fine-tuning is expensive and can cause "catastrophic forgetting," where fixing one bug breaks unrelated features.

A new alternative is weight-space programming, where an LLM agent inspects a model's internal activations and weights, identifies the specific units responsible for a concept, and writes a targeted weight change to fix the behavior.

The problem? Standard models are trained only for accuracy, not "editability." Their internal representations are often "entangled," meaning a simple weight change to fix one concept might have massive, unpredictable side effects. This paper asks: Can we train models to be inherently easier for LLMs to edit?

Methods

The authors developed a two-part solution: a testing framework called WeightLab and a training method called Edit-Ready Training (ERT).

1. WeightLab Sandbox

WeightLab is a restricted environment where LLM agents (like GPT-6 Astra or Claude Opus 5.5) are given a "budget" of forward passes (probes), weight reads, and weight edits. They cannot use gradients or traditional training; they must act like engineers manually debugging a circuit.

2. Edit-Ready Training (ERT)

ERT prepares a network for future edits by simulating the LLM's behavior during the model's initial training.

  • Edit Simulation: During each training step, the system picks a concept and simulates a "request" to move it to a different class. It applies a simple, rank-one weight update to the classifier head based on the contrast between "support" images (the concept to change) and "reference" images (everything else).
  • Edit Loss: The model is then penalized if this simulated edit fails to change the target concept or if it messes up the predictions for unrelated reference images.
  • Backpropagation: Crucially, the errors from these simulated edits are backpropagated through the entire network. This forces the backbone to learn features that are "clean" and easily shifted by simple linear edits later on.
Figure 1. Training vision models for weight-space programming. (a) Edit-Ready Training (ERT) simulates edits during training to make vision models better editable by LLMs. (b) Through our WeightLab an LLM agent can iteratively edit a vision model’s weights to change its behavior. (c) ERT improves vision model editability through LLMs, which is indicated by an increased mean net edit score over all nine settings, and the gain of ERT over ERM on the three settings that are hardest (bars: 95% bootstrap confidence; ∗: p < 0.05).

Datasets

The researchers evaluated their approach across nine different settings using three backbones (ResNet-18, ResNet-50, and ViT-B/16) and four datasets:

  • Waterbirds: Focused on fixing reliance on spurious background correlations.
  • CIFAR-100: Used for fine-grained relabeling and fixing class confusions.
  • PACS: Focused on "subgroup fixes," such as correcting misclassified sketches.
  • Entity-30: A 30-class subset of ImageNet used for complex confusion repairs.

Key Insights and Ideas

  • LLMs are "Rank-One" Editors: Analysis of thousands of edit trajectories revealed that frontier LLMs naturally prefer making sparse, rank-one changes to the classifier head. They look for a "direction" in feature space that separates the target concept from others and shift the weights along that axis.
  • Entanglement is the Barrier: Standard training (ERM) often results in features that are "smeared" across many units. ERT encourages "disentanglement," ensuring that a concept is represented by a few distinctive units or a clean direction that a simple edit can isolate.
  • Preparation is Key for "Hard" Tasks: In easy scenarios, standard models are already somewhat editable. However, in "hard" settings where standard models resist manual editing, ERT provides a massive boost in success rates.
  • Backbone Shaping: Even though LLMs mostly edit the head of the network, ERT's benefit comes from how it shapes the backbone features to be more "ready" for those head edits.

Key Results

The study involved over 2,600 edit processes and 75,000 tool calls. ERT emerged as the most effective preparation method:

  • Highest Mean Scores: ERT achieved the highest mean "net score" (success rate minus side effects) for both GPT-6 Astra (0.74) and Claude Opus 5.5 (0.72), outperforming standard training and other "editable" baselines like ENN.
  • Massive Gains in Hard Settings: On the Entity-30 confusion task, ERT raised Astra’s score from 0.46 to 0.79. On Waterbirds with a ViT backbone, Claude’s score jumped from 0.33 to 0.76.
  • Efficiency: LLM agents typically reached their peak performance within just a few edit calls, especially on ERT-prepared models.
Progress of net score and accuracy over multiple edit calls

Limitations and Future Work

  • Overall Accuracy Trade-off: While ERT makes models easier to fix, the actual process of editing still results in a 3-4% drop in overall test accuracy. Future work needs to find ways to preserve general performance during targeted edits.
  • Task Scope: The study focused on image classification. Whether these techniques translate to more complex vision tasks (like detection) or generative models remains an open question.
  • Backbone Edits: Currently, LLM agents rarely touch the backbone weights. The authors suggest that future training could rehearse edits deeper in the network to enable more fundamental model repairs.

Summary updated

Abstract

It has been demonstrated that, in certain settings, large language models can perform weight-space programming to directly modify the weights of neural networks. However, it is unknown how dedicated training of these networks can contribute to the successes of weight-space programming. We present a benchmark of LLM-driven weight-space programming on vision models, in which five LLMs edit convolutional networks and vision transformers on relabeling and error-correction tasks on Waterbirds, CIFAR-100, PACS and Entity-30. We report more than 2,600 such edit processes, whereby each edit is scored on held-out images for success, side effects and accuracy loss. We compare standard training with label smoothing, gradient-based editable training, and our Edit-Ready Training (ERT), a novel method that rehearses the direct, closed-form weight edits LLMs perform. We show that preparing networks for editing raises the success of LLM edits substantially where standard networks are hard to edit. ERT further achieves the highest mean score for GPT-6 Astra and Claude Opus 5.5 across all evaluated baselines. We will release our LLM testing framework alongside all tasks, editing trajectories and scores. Project page: https://viscom.uni-ulm.de/wsp
BibTeX
@misc{ropinski2026training,
  author = {Timo Ropinski and Sebastian Kranz},
  title = {{Training Vision Models for LLM-Driven Weight-Space Programming}},
  year = {2026},
  eprint = {2611.26408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26408},
}
Figure